Paper: Faithful and Efficient Explanations for Neural Networks via Neural Tangent Kernel Surrogate Models (ICLR-2024)

Reviewer comment (Theory):
In section 5, you mentioned: "...an interesting follow-on work would investigate using kernel functions in K-Nearest Neighbors surrogate models." How much argument of this paper can transfer to KNN or generally any other surrogate models on explaining NN decision?

Author response:
2. Re:  In section 5, You mentioned: "...an interesting follow-on work would investigate using kernel functions in K-Nearest Neighbors surrogate models." How much argument of this paper can transfer to KNN or generally any other surrogate models on explaining NN decision?
    * An excellent question that we invite the reviewer to discuss with us. We have conducted some preliminary work investigating the roles other surrogate models (kernel-SVM and kernel-KNN) can play in investigating the NN model. So far we understand that the choice of surrogate model can have profound impact on what we might infer. For example, with the kernel-KNN we would specifically be modeling that only the local structure of the feature space matters to the classification; in effect, assuming sparsity a priori. However, it is not yet clear how we should assign a class probability/logit value to utilize the Kendall-$\tau$ measure. We do include some analysis with kernel-SVMs as well looking into adversarial examples. We infer that kSVMs are not a good surrogate for modeling the effect of adversarial examples since the robustness scales differently between the NN and the kSVM. This may be in part due to the structure of the kSVM itself: in the kSVM, only a sparse number of data points near the decision boundary matter for classification (i.e., the support vectors). Perhaps it is easier for an adversarial example to fool the kSVM given fewer samples the kSVM is utilizing than the NN (under our hypothesis that the NN is not sparse).

Reviewer comment (Writing):
Section 2 PRELIMINARIES Neural Networks for Classification third line: it should be \mathcal{Y} instead of Y.

Author response:
3. Re: also, there are some of the minor typos in the paper
    * We thank the reviewer for pointing out the mistakes and will fix them in the revision

Reviewer comment (Evaluation):
Appendix F FORMAL DEFINITION OF EVALUATION METRICS last equation: it should be SS_res instead of SS_ret.

Author response:
3. Re: also, there are some of the minor typos in the paper
    * We thank the reviewer for pointing out the mistakes and will fix them in the revision

Reviewer comment (Novelty):
The paper should be more clear about its relative contribution to the literature (and what its main contribution is). The paper gives confusing statements about what is new and what is taken from the literature. The authors state that the 3 main contributions are (1) new kernel functions for NTK approximation, (2) first to show that eNTK models are consistently correlated with NNs across experiments, (3) first to compare NNs decisions through NTKs through a data attribution and explain by example strategy.

Author response:
2. Re: Contributions are unclear. For point (1) however the authors also state that the trNTK was introduced in Chen et al. 2021 (end of page 2)...  
    * The full story is somewhat complex and in editing to meet page limits it became less clear. Chen et al. (2021) never formally defined the trNTK and never mentions it in their main body. They only describe trNTK in a single sentence in the appendix: (Appendix G, first sentence: "We approximate the neural tangent kernel on the i1k (1/10) subset by averaging over block diagonal entries ... in the full NTK."). Therefore, we feel our wording is accurate albeit slightly confusing: we are the first to formally define the trNTK. In Chen's work, the trNTK was a (hidden) computational convenience to enable their experiment for a single data set; in contrast, our work treats the trNTK to be the central object of study as an approximation of the NTK. We believe that this is a novel contribution but at the same time we should give proper credit to Chen et al. We will look to revise the contribution section and introduction to make this point clearer.

Reviewer comment (Novelty):
For point (1) however the authors also state that the tr NTK was introduced in Chen et al. 2021 (end of page 2) and that the random projections approach is based on Park et al. 2023 (end of page 1). Is the main contribution of the paper proposing a new NTK method or evaluating empirical exercises?

Author response:
4. Is the main contribution of the paper proposing a new NTK method or evaluating empirical exercises?  
    * We view the paper as having three major contributions: a) We describe methods, building off recent successes from TRAK, that use projection matrices to compute efficient approximations to the eNTK. b) We establish the faithfulness metric Kendall-$\tau$ and detail how it is superior to previous methods relying on test accuracy. c) We identify the trNTK kernel function as the best performing surrogate model to the NN across a variety of experiments. Analyzing this choice of kernel in a data attribution context, we find that many training data points influence the output on any new data input.

Reviewer comment (Experiments):
The authors consider different alternative NTKs to compare to the trNTK, but never compare in the main text the methods to the eNTK or the pNTK. Given that the motivation of the paper in the abstract, introduction etc is to approximate the eNTK it is odd that this is not done in the main text of the paper. While computational constraints are important, maybe it could be done on a simpler dataset (MNIST)?

Author response:
5. Re: The authors consider different alternative NTKs to compare to the trNTK, but never compare in the main text the methods to the eNTK or the pNTK...  
    *  Our goal was to develop surrogate model methods that scale to large computer vision experiments and transformer models with potentially large number of classes. This implies we will need to find an approximation to the eNTK as the eNTK scales quadratically with the number of classes. A direct comparison to the eNTK is an important topic, but we did not consider it part of this paper's scope.  
    * We understand the reviewer's concern that we did not compare with the pNTK. Our rational is that we "brute-forced" the computation of several trNTK and compared them with the proj-trNTK. After observing that the proj-trNTK had exponentially decaying residuals (Appendix E) with the trNTK, we felt secure in evaluating the proj-pNTK alone.

Reviewer comment (Novelty):
For point (2) if using the rank correlation is new it should be clearly stated as a major contribution. The paper repeatedly expresses that other measures are flawed and while the authors give some reasons why, without a proper theoretical statement the authors should at least relate these notions more directly to the findings of the empirical examples. For example, what is a clear case in which using fit or pearson correlation would be misleading in the sense that two NTK models give you very different attributions despite having the same fit to the NN, but rank correlation is not misleading.

Author response:
6. For point (2) if using the rank correlation is new it should be clearly stated as a major contribution. The paper repeatedly expresses that other measures are flawed and while the authors give some reasons why, without a proper theoretical statement the authors should at least relate these notions more directly to the findings of the empirical examples. For example, what is a clear case in which using fit or Pearson correlation would be misleading in the sense that two NTK models give you very different attributions despite having the same fit to the NN, but rank correlation is not misleading.  
    * We copy from our comment to all for the reviewer's convenience:  
    * To begin: Pearson correlation measures the ratio of the covariance between a set $\mathcal{X}$ and $\mathcal{Y}$ to the product of the square root of the variance in each set. Kendall-$\tau$ measures the degree to which the sets are monotonic with one another, irrespective of the functional form of any mapping between sets.  
    * Consider the following toy scenario: imagine there are two independent classification models that achieve nearly 100\% accuracy. Furthermore, the output of these models is a normally distributed random variable with $\mathcal{N}(\mu_1,\sigma_1)$ if the true label is 1 and $\mathcal{N}(\mu_0,\sigma_0)$ if the true label is 0. If $\sigma_0,\sigma_1$ are small compared to $\mu_0-\mu_1$, then the Pearson correlation will be nearly 1.  
    * Intuitively, this is because this describes point clouds centered at $\mu_0$ and $\mu_1$, and if those point clouds have a high degree of separation relative to their variance each random variable has in each point cloud ($\sigma_0$ and $\sigma_1$) then the covariance will be nearly equal to the variance of each random variable. This means that the Pearson correlation will be 1. We want a measure of the dependence between two random variables, and we have now shown that Pearson can result in value 1 even if the variables are explicitly not dependent on one another.  
    * In contrast, Kendall-$\tau$ returns a lower bound value of 0.5 in this toy model (assuming equal class sizes). So while Kendall-$\tau$ is still sensitive to the failure mode affecting Pearson correlation, with his probability Kendall-$\tau$ does not reach its maximum, so can still be a useful tool to distinguish between independent and dependent functions.  
    * While simple, this thought experiment is revealing. We have two point clouds at $\mu_1$ and $\mu_0$. In Appendix I.1 (Figure 7) we can see that the point cloud structure of the logit observations mimic this case. To help explain this more concretely, we provide a Jupyter notebook file with these experiments in the supplementary work to demonstrate our arguments.

Reviewer comment (Experiments):
For point (3) the paper should explain more carefully why these are carried out. If the goal is to assess how good a NTK approximation is by considering whether it performs similarly to the NN in a data attribution task then this should be the focus of the results. It seems that the authors do these exercises in the appendix, but in the main write-up they just give an instance of this and it is unclear how much we can learn from it. If well addressed I would be inclined to raise my score.

Author response:
7. Re: For point (3) the paper should explain more carefully why these are carried out. If the goal is to assess how good a NTK approximation is by considering whether it performs similarly to the NN in a data attribution task then this should be the focus of the results. It seems that the authors do these exercises in the appendix, but in the main write up they just give an instance of this and its unclear how much we can learn from it. If well addressed I would be inclined to raise my score.  
    * The qualitative analysis we provide between kernels is an important tool to help build intuition and formulate new hypotheses. For example, the framework and visualizations were critical in allowing us to reach the conclusion that the NN's predictions depend more so upon all the training data rather than a sparse number of exemplars, since the NN has the highest correlation with the trNTK. This faithfulness allows us to infer non-sparsity since the outliers of attribution from the trNTK do not dominate the total attribution. This serves as a counter-narrative to the prevalent explain-by-example paradigm which has assumed sparsity a priori.  
    * As an example of how we can utilize these tools, see Figures 20, 31, and 42: In Figure 20, we see that the mean plane and horse attribution score are highest driving the classification in their respective logits, but with the mean attribution from the remaining classes having higher negative impact on the plane logit. Figure 31 zooms into the horse logit for the trNTK, showing that the logit horse value is dependent upon the horse class. Even though our analysis shows that a sparse number of representations do not best describe the NN predictions, we can still leverage the most similar data points to help us form hypothesis of NN behavior. In Figure 42, we show the most similar images are of people riding horses. Upon seeing this, we realized that we had not noticed that the initial test image is actually of a person siting "on top" of an airplane! While we still must find a way to test this hypothesis, we believe that helping researchers form these hypotheses is still a valuable tool in understanding misclassification.

Reviewer comment (Presentation):
The paper may also suffer from typos and plots are sometimes misleading (squished axis in Figure 1). Some typos include pseudo vs psuedo, missing points, figure labels that overlap, subscripts in mathematical notation etc. Some references are also repeated.

Author response:
8. Re: The paper may also suffers from typos and plots are sometimes misleading...  
    * We thank the reviewer for pointing out these mistakes and will correct them in the revision.

Reviewer comment (Evaluation):
Only the rank correlation of the softmax probabilities for the **correct** class is considered. However, to be faithful enough, the surrogate model should also behave similarly to the NN for the **incorrect** classes. An important application of data attribution is to explain why a NN makes a wrong prediction. This is not considered in the paper.

Author response:
Re: Only the rank correlation of the softmax probabilities for the correct class is considered...
    * We agree with the reviewer that a current limitation of Kendall-$\tau$ is that it does not consider the logit value of incorrect classes. We think that an ideal faithfulness measure would include this information,  we will add a discussion of Kendall-$\tau$ the limitation of Kendall-$\tau$ to our limitation section; however, we feel despite our use of Kendall-$\tau$ is a significant step away from previous works that rely on test accuracy to measure the similarity of surrogate functions to the neural network. 
Re: ... However, to be faithful enough, the surrogate model should also behave similarly to the NN for the incorrect classes.
    * We agree with the reviewer's intuition. While this set of experiments have not appeared in the paper, we conducted an investigation into a "misclassification coincidence rate" as an additional line of evidence suggesting that the kGLM can mimic the behavior of the NN on misclassifications. The misclassification coincidence rate, $R_{\textrm{miss}}$, is the ratio of the number of times the kGLM and NN predict the same class out of all times either are incorrect over the number of times either the NN or the kGLM are incorrect
    \begin{equation}
    R_{\textrm{miss}} = \frac{ |\{ i : \textrm{kGLM}(\mathbf{x}_i) = \textrm{NN}(\mathbf{x}_i) \ne y_i \}| }{ | \{i : \textrm{kGLM}(\mathbf{x}_i) \ne y_i\} \cup  \{i : \textrm{NN}(\mathbf{x}_i) \ne y_i\}| }
    \end{equation}
    * This rate is actually very high (about 75\% coincidence of misclassifications on ResNet18 for trNTK), which is powerful evidence suggesting that the kGLM can mimic the NN overall for both correct classification and misclassifications. We will include this analysis in the revision.
Re: An important application of data attribution is to explain why a NN makes a wrong prediction. This is not considered in the paper.
    * While our main body is primarily concerned on establishing our framework for data attribution and then an initial analysis re. the assumption of sparsity of the attribution scores, our framework can be extended to study misclassification. Our current status is to try and understand holistically from all the visualization generated why the NN made a misclassification by investigating the kGLM as proxy. Take as example figures 20, 31, and 42: In figure 20, we see that the mean plane and horse attribution score are highest driving the classification in their respective logits, but with the mean attribution from the remaining classes having higher negative impact on the plane logit. Figure 31 zooms into the horse logit for the trNTK, showing that the logit horse value is dependent upon the horse class. Even though our analysis shows that a sparse number of representations do not best describe the NN predictions, we can still leverage the most similar data points to help us form hypothesis of NN behavior. In figure 42, we show the most similar images are of people riding horses. This was quite the realization, since we had not noticed that the initial test image is actually of a person siting on top of an airplane! While we still must find a way to test this hypothesis, we believe that helping researchers form these hypothesis is a valuable tool in understanding misclassification.

Reviewer comment (Theory):
Eq. (4) is confusing. In the denominator, the $\cdot ^ {\frac{1}{2}}$ is applied to the inner product. However, according to Appendix C and the definition of cosine similarity, the $\cdot ^ {\frac{1}{2}}$ should be applied to the sum, not the inner product.

Author response:
Re: Eq. (4) is confusing. In the denominator, the sqrt is applied to the inner product. However, according to Appendix C and the definition of cosine similarity, the sqrt should be applied to the sum, not the inner product.
    * We thank the reviewer for pointing out mistake; we will add parenthesis to Eq (4) and to ensure the equation matches the definition of cosine similarity.

Reviewer comment (Presentation):
The quality of Figure 2 could be improved.

Author response:
Re: The quality of Figure 2 could be improved.
    * We agree; there are rendering errors that we will fix in the revision and double check that boxes do not move in front of the axis labels.

Reviewer comment (Evaluation):
Is the non-sparsity of the attribution a general phenomenon or just a property of trNTK? Is it a consequence of the statement "It has been suggested that this normalization helps smooth out kernel mass over the entire training dataset"?

Author response:
Re: Is the non-sparsity of the attribution a general phenomenon or just a property of trNTK? Is it a consequence of the statement "It has been suggested that this normalization helps smooth out kernel mass over the entire training dataset"?
    * We thank the reviewer for the interesting question. To be succinct: it appears so. We provide some visualizations relevant to this question in figures 21 - 31 in the appendix. There, we compare the trNTK to the un-normalized trNTK. In the unnormalized trNTK we observe that the bulk of attributions are centered nearly on zero with a very small IQR, while the normalized trNTK has broader IQR and positive values mean attribution. This suggests that the unnormalized trNTK is in fact using a few exemplars in its kernel machine to make its classification (i.e., a few exemplars make up the majority of the attribution mass in the unnormalized NTK). Our non-sparsity result is a consequence of the fact that the faithfulness of the cosine-normalized trNTK as measured by Kendall-$\tau$ is higher (substantially so for some models) than the un-normalized trNTK.

Reviewer comment (Evaluation):
Content: Surfacing similar images is a not a meaningful evaluation of attribution. It is a good sanity check, but doesn't say anything about surrogacy. For example, finding similar images using CLIP similarity would also show similar images, though CLIP is in no way a "surrogate" to the model being studied.

Author response:
Re: Surfacing similar images is a not a meaningful evaluation of attribution.
    * We wholeheartedly agree with the limitations of surfacing similar images as the reviewer states --- in fact, this is a major conclusion we draw in the paper. See the end of section 5 paragraph 1: "Therefore, presenting the top highest attribution training images without the context of the entire distribution of attribution is probably misleading." Our kernel machine surrogate framework goes beyond simply surfacing similar images to show how the surrogate model, which is deeply correlated with the underlying neural network, leverages **all of the training data** to make classification on new test data.

Reviewer comment (Evaluation):
More broadly, I'd be more careful about making any claims of "data attribution" (which has a specific meaning as used in recent ML) as the paper does carry out any counterfactual evaluations.

Author response:
Re: I'd be more careful about making any claims of data attribution ... as the paper does carry out any counterfactual evaluations.
    * We acknowledge the historical context of data attribution in classical statistics, and recent research in data attribution that have utilized counterfactual experiments; however, it is not the only sense in which "data attribution" is used. We subscribe to the framework given in Park et al 2023 (TRAK): "a data attribution method computes a score for each training input indicating its importance to the output of interest." Specifically, they cite the approach of Yeh et al 2018 (i.e. Representer Points) as a form of data attribution, and we mirror the same sense of data attribution in our work. We describe how we compute our data attribution score in great detail in the comment to all reviewers.

Reviewer comment (Novelty):
Overall, the contributions seems somewhat marginal. Also, the fast approximate versions implemented primarily rely on prior work (Park et al.)'s implementation, so not sure there is much to claim as contribution there (since Park et al. also used it for faster approximations to eNTK).

Author response:
Re: Overall, the contributions seems somewhat marginal. Also, the fast approximate versions implemented primarily rely on prior work (Park et al.)'s implementation, so not sure there is much to claim as contribution there (since Park et al. also used it for faster approximations to eNTK).
    * We politely but strongly disagree: Park et al. (TRAK) neither compute the eNTK nor any approximation of the eNTK. While Park et al. do reference the eNTK and devote a section to show how it is a part the TracIn kernel, the actual calculation of the eNTK itself is not implemented in their code. You can view the relevant lines of code from TRAK [here](https://github.com/MadryLab/trak/blob/main/trak/modelout\_functions.py\#L84]).  Notice that the model output functions implemented are only the loss, rather than the neural network function itself. We have been in direct communication with the maintainer of TRAK to bring the capabilities we describe in this work to the TRAK software package.
    * It is true that our fast approximation would not be possible without TRAK (and so  TRAK rightfully takes a prominent position in our paper); however, building off each other's work is a success story of the open-source community and is at the foundation of AI research. We see this as a major strength rather than a weakness.

Reviewer comment (Writing):
Writing: is hard to follow at times and doesn't provide the relevant details (see Questions). On one hand, the paper goes into more detail than necessary in defining rank correlation / R2, etc from scratch, and at the same time, doesn't actually provide details about what those measures are computed over exactly. It's possible I missed it, but at least doesn't seem very clearly written based on my multiple attempts to parse this information.

Author response:
Re. Writing ... the paper goes into more detail than necessary in defining rank correlation / R2, etc from scratch...
    * We thank the reviewer for pointing out the confusion. We agree that the space devoted to the Kendall-$\tau$ definition could be saved by moving it to the appendix. We would instead use this space to discuss the motivation for the Kendall-$\tau$, which we believe is more important than discussing why Pearson is flawed. If it interests the reviewer, we provide additional information in the comment to all reviewers about our argument for why we use Kendall-$\tau$.

Reviewer comment (Evaluation):
Confused by what the rank correlation is measured over exactly. I understand it's measured between the truth model outputs and surrogate model outputs, but what is it varied over? Are you measuring across different inputs x?

Author response:
Re: Confused by what the rank correlation is measured over exactly. I understand it's measured between the truth model outputs and surrogate model outputs, but what is it varied over? Are you measuring across different inputs x?
    * Yes: we are varying it over the different model inputs x from the test dataset. For cases beyond binary classification (single logit), we also must make a choice for which output of the neural network/ surrogate model to use. We chose to use the output corresponding to the correct class (as given by the ground-truth labels).

Reviewer comment (Presentation):
A bit confused by what the message/takeaway of the box/distribution plots are. Can the authors elaborate?

Author response:
Re: A bit confused by what the message/takeaway of the box/distribution plots are. Can the authors elaborate?
    * Following are the key takeaways. 
    * In comparison to explain-by-example techniques that surface a few images, we demonstrate a visualization that shows how we use the distribution of all attribution scores. This is important because by considering the entire distribution we have not made the assumption that attribution is sparse a priori. 
    * We want to evaluate whether the sparsity assumption is valid. Given that the mean of the attribution score falls within the interquartile range across all training datapoints, we know there is not an extremal point attribution significantly moving the average. In effect: we know that none of the outliers are dominating the attribution, but rather, the bulk properties of attribution from all training points are responsible for the model output.

Reviewer comment (Presentation):
It seems that the eNTK is only defined in Appendix D, so it's a bit hard to contextualize pNTK and trNTK when they are introduced.

Author response:
Re: It seems that the eNTK is only defined in Appendix D, so it's a bit hard to contextualize pNTK and trNTK when they are introduced
    * We thank the reviewer for pointing this out, and in the revision we will add an inline equation to define the eNTK to provide some additional context for the reader.

Reviewer comment (Theory):
The paper relies on previous work to establish credibility of attribution-based scores for neural network explanation. It doesn’t seem obvious that attribution is the same as similarity for learned kernel functions.

Author response:
Re The paper relies on previous work to establish credibility of attribution-based scores for neural network explanation.
    * We have expanded upon our argument to establish the credibility of our attribution scores in the comment to all (point 2). We resolve to present this in the revision so our work is more self-contained.
Re It doesn’t seem obvious that attribution is the same as similarity for learned kernel functions.
    * In Eq (3) we describe how attribution is the product of the kernel similarity with the learned weights, so is \textbf{not} just the similarity. We expand upon this in the comment to all reviewers as well (point 3), since it was a shared confusion. We will be more explicit in our revision.

Reviewer comment (Writing):
I find the second sentence in the abstract confusing. I expected this trend to have to do with using kernel-based models for data attribution rather than to “investigate a diverse set of neural network behavior”. Isn’t the goal of your paper exactly to apply kernel models to investigate network behavior?

Author response:
Re I find the second sentence in the abstract confusing. I expected this trend to have to do with using kernel-based models for data attribution rather than to “investigate a diverse set of neural network behavior”. Isn’t the goal of your paper exactly to apply kernel models to investigate network behavior?
     * Our explicit goal is to use kernel methods as a surrogate for neural networks, and the application we focus on in this paper is data attribution. We agree that our attempt to remain general in the second sentence of the abstract reads overly pedantic for the sake of the narrative device used in the third sentence. We will address this sentence in the revision.

Reviewer comment (Evaluation):
The 3rd experiment on qualitative evaluation of attribution is weak. A user study is probably beyond the scope of this paper, and I believe the work is strong enough to stand without such a study. However, the paper would significantly benefit from some discussions about how these attributions could be better qualitatively evaluated in the future.

Author response:
Re The 3rd experiment on qualitative evaluation of attribution is weak. A user study is probably beyond the scope of this paper, and I believe the work is strong enough to stand without such a study. However, the paper would significantly benefit from some discussions about how these attributions could be better qualitatively evaluated in the future.
    * We share a preference for stronger quantitative evaluation as well; however, we believe the qualitative evidence we have collected is still important to share because researchers often generate new hypotheses from looking at the data in a new qualitative way.
    * We seek to publish this result without human-subjects testing. We will add a short discussion of the limitations of explainability and caution the reader that a user-study must be performed in the revision.

Reviewer comment (Writing):
The paper never explicitly defines the empirical NTK in its own notation. Could you add this prior to defining the trNTK or pNTK in order to allow an easier discussion of the approximations introduced?

Author response:
Re: The paper never explicitly defines the empirical NTK in its own notation. Could you add this prior to defining the trNTK or pNTK in order to allow an easier discussion of the approximations introduced?
    * We will add in the eNTK equation for the revision.

Reviewer comment (Presentation):
The take-away from Figure 2 is not exactly clear. Is this just meant to show that attribution scores are not sparse?

Author response:
The take-away from Figure 2 is not exactly clear. Is this just meant to show that attribution scores are not sparse?
    * Figure 2 is meant to convey the following ideas a) how we do not assume attribution is sparse a priori, so we look at the attribution distribution from all training points; b) because we do not assume sparsity, we can check whether sparsity emerges naturally. We do not find any evidence supporting the assumption of sparsity since the outliers are not extreme enough in comparison to the bulk mass of attribution in each logit for the trNTK.

Reviewer comment (Writing):
Your Chen ICML’22 reference is duplicated. Did you intend to cite two different papers?

Author response:
Re: Your Chen ICML’22 reference is duplicated. Did you intend to cite two different papers?
    * We thank the reviewer for pointing out the mistake and will fix in the revision. We did not intend to cite two different versions of the same paper.

Reviewer comment (Writing):
The notation is non standard. Most papers use $y$ not $z$ for ground-truth labels. I can see this causing some readers mild confusion.

Author response:
Re: The notation is non standard. Most papers use y not z for ground-truth labels. I can see this causing some readers mild confusion.
    * We wanted to be very clear to distinguish between a logit and a probability vector; since, we expected many readers would confuse whether the final softmax/sigmoid activation function are being included in our NTK computation. This motivated our use of X$\rightarrow$Y$\rightarrow$Z since the alphabet order matches the order of NN$\rightarrow$logit$\rightarrow$probability vector. We will emphasize the break from standard notation in the revision.

Reviewer comment (Theory):
Have you considered using the kernel function directly to evaluate sample similarity? Why do you choose to include the weights from the kernel machine in all attribution scores?

Author response:
Re: Have you considered using the kernel function directly to evaluate sample similarity? 
    * We have, yes. While the weights are used in our definition of data attribution, we actually do not use the weights to surface the most similar images, (figures 31-41, in Appendix J.2). We observed that visualizing the most similar vs the highest attribution images often result in qualitatively more shared concepts in the image. For example, if the test image is a picture of a horse's head looking left, the most similar training images will be of horses looking left; while the highest attribution images might more varied pictures from the horse class.

Reviewer comment (Reproducibility):
It is not totally clear how the Kendall-$\tau$ statistic is computed. A couple of sentences would make this portion more reproducible. Do you take the matrix of all the logits produced on a test set ($N x C$) for the NN and for the surrogate model, flatten these two matrices into vectors, and compute the rank correlation? Is the rank correlation computed per-test-output and averaged over the test set?

Author response:
Re: It is not totally clear how the Kendall-$\tau$ statistic is computed...
    * We apologize for the confusion. We compute the Kendall-$\tau$ over the vector of created from correct class logit values (given by the ground truth labels) of the kGLM and NN, evaluated on the entire test dataset.

Reviewer comment (Experiments):
Why is trNTK initially introduced with cosine normalization and projNTK not? Don’t your experiments include cosine normalization for all kernels?

Author response:
Re: Why is trNTK initially introduced with cosine normalization and projNTK not? Don’t your experiments include cosine normalization for all kernels?
    * We think the reviewer might be referring to the pNTK instead of either the proj-pNTK or proj-trNTK? We introduce the pNTK without the normalization because we mean to reproduce the pNTK exactly as from Mohammadi et al. 2022. We do not perform any experiments on the pNTK. When we compute the proj-pNTK we do apply the cosine-normalization. **We use the cosine-normalization for all kernels except for the unnormalized-trNTK, or** $\text{trNTK}^0$

Reviewer comment (Theory):
Is there any theoretical statement you can add about the variance of the projection-based kernel estimates, e.g. based on JLS? The choice of 10240 dimensions seems arbitrary and model-dependent.

Author response:
Re: Is there any theoretical statement you can add about the variance of the projection-based kernel estimates, e.g. based on JLS? The choice of 10240 dimensions seems arbitrary and model-dependent.
    * We assume that the reviewer is referring to the Johnson-Lindenstrauss Lemma. On first inspection, we believe it should be straightforward to use the JL Lemma to  show that each inner product in the original and projected kernels are close (in a multiplicative sense, if appropriately normalized). This in turn implies that the (empirical) distributions of entries in the NTK and projection-NTK are close. Because the projection dimension can be chosen to make this bound small, we can partially control the closeness of various statistics of the distributions (including the variance) by choosing the projection dimension. If the reviewer thinks it will better justify the choice of projection dimension, we can try to add (or provide a sketch) of such a result in the final version.
    * In the submitted version of the manuscript, we treated the projection dimension as a hyperparameter selection task specific to each model, where there is a trade-off in the projection dimension size between compute time and variance of the residual distribution. For our resources, 10240 created a reasonable compromise for both Bert and ResNet18, but we will be more explicit in the revision that this choice is of simple convenience.

Reviewer comment (Evaluation):
Does the introduction of cosine normalization explain the experimental result “that the highest attributed images from the trNTK (and furthermore all evaluate kernel functions) have relatively small mass compared to the bulk contribution, suggesting that the properties of the bulk”?

Author response:
Re: Does the introduction of cosine normalization explain the experimental result “that the highest attributed images from the trNTK (and furthermore all evaluated kernel functions) have relatively small mass compared to the bulk contribution, suggesting that the properties of the bulk”?
    * We believe this comment is correct: by comparing the trNTK and $\text{trNTK}^0$ we see that the $\text{trNTK}^0$ has much smaller scale of IQR to the most extreme data attribution points relative to the trNTK's. This result is highly suggestive that the cosine normalization helps the trNTK smooth out the attribution amongst all training points. This fact is not reflected in the quoted statement in the reviewer's question, so we will modify it in the revision.