Paper: Exact Path Kernels Naturally Decompose Model Predictions (ICLR-2024)

Reviewer comment (Theory):
The OOD is obtained by analyzing the span of the linear subspace formed by the loss gradient vectors. However, the analysis may not take into account the nonlinearity.

Author response:
We understand your concerns about the limitations of linear methods to address the nonlinear nature of neural networks. We have substantially revised section 4 in order to discuss these details in a clearer way.  In both of our applications, the nonlinearity of the model is contained in how the model spatially transforms and represents the data for the task. Both in parameter space and in input space, this generates nonlinear (curved) spatial arrangements of data. Although this decomposition can be used to analyze nonlinearity, our applications to both OOD and dimension measurement are limited to the tangent space around specific individual data. In this context, the tangent space is linear. Our revised section 4 addresses the implications of our choice to pick a linear basis for the final trained state and extend that basis across all training steps. An exact decomposition of a prediction into contributions for each training point requires a non-linear basis across the space of training paths. However, since the training path is fixed, we can still explain all variation by only looking at the linear subspaces for each training step. This is subtle but useful and relies on the fact that the training gradients in this representation do not depend on $t$. We justify reducing this down to only a truncated basis for the final training step after Corollary 4.2. We hope that these revisions address your concerns and questions, and we greatly value your continued feedback to continue clarifying these subtle details.

Reviewer comment (Theory):
What's the significance of \epsilon in Theorem 3.1 and how is it determined?

Author response:
In response to your first concern, you are correct, a definition for $\varepsilon$ was missing. In this case $\varepsilon$ represents the learning rate hyperparameter of the neural network and we have added a comment defining it as such to the definitions in Theorem 3.1.
In regards to $\theta_s(t)$ the definition of this value can be found in section 3 equation 7 and the relevant properties of $\theta_s(t)$ as it relates to $\theta_s$ are defined in equations 5 and 6. We agree that the notation of this section may be confusing, so we have clarified the domain of $t$ and added a section explaining the properties of $\theta_s(t)$.

Reviewer comment (Novelty):
I find that there is repetition like Bell et al. on Section 3, remark 1 and remark 2. That is, Bell et al. make the similar remarks on SGD and subsampling. Proving techniques resembles the approach in Bell et al.

Author response:
Thank you for your thoughtful feedback on our manuscript. We appreciate you taking the time to provide an in-depth review and value your expertise in improving the quality of our work.

We understand your concerns about the limitations of the gEPK and its similarity to prior work. Per your recommendations, we have made revisions to improve this aspect in the beginning of section 3. We now more clearly indicate the differences between work done previously by Bell et al. and our paper. In addition we have revised Section 4 in order to provide more detail on the novelty of our approach.

Thank you again for your help in improving the quality of this work. If you have any further questions or concerns we would be happy to continue this discussion.

Reviewer comment (Novelty):
While EPK gives a representation of test points in terms of a vector space where the basis is computed from gradients at every training step, most OOD methods consider the optimized parameter which is $\theta_S$ as the notation in the paper. This means that the existing OOD does not consider the parameter trajectories of optimization.

Author response:
Thank you for your thoughtful feedback on our manuscript. We appreciate you taking the time to provide an in-depth review and value your expertise in improving the quality of our work.

Regarding your concern of differences between accounting for the training path or evaluating only at the final trained state, we have addressed this issue in the revised section 4. You will now find Theorem 4.1 which refers to the most general basis for all training parameter gradients along the path and Corollary 4.2 which discusses the use of a reduced basis and discusses a justification for using only the final training state. This provides a more rigorous foundation for gradient based OOD by taking into account this full decomposition, and also places many cutting edge OOD approaches in a more rigorous and interpretable context by highlighting their relation (as truncations or as implicit subspaces) to the subspaces and basis we define. There is more work to be done on this in order to bring it into a practical setting. In response to your concern we have added more discussion surrounding this topic to the conclusion of this paper.

Thank you again for your help in improving the quality of this work. If you have any further questions or concerns we would be happy to continue this discussion.

Reviewer comment (Novelty):
My main concern is that the theoretical results seem to follow directly from the arguments of Bell et al., 2023. Perhaps it could help if the authors added a discussion on the technically novel aspects of the analysis, or alternatively emphasized more on the applications of this decomposition as the main contribution.

Author response:
- Substantially revised Section 4 for clarity and to highlight our contributions to the detailed theory needed to use this method to decompose projections. 

The focus of this paper is on the use of a generalized exact path kernel to decompose predictions. In order to present this theoretical framework in a readable, complete, and transparent way, we have included a proof of the generalized representation which is derived from Bell et al. This is not our main contribution. Our main theoretical contribution is in the application of this representation to decompose predictions and the consideration of many details including how this decomposition applies across training steps and model layers. We have presented these details alongside applications to OOD and manifold dimension measurement to demonstrate the applicability and usefulness of this approach.

Reviewer comment (Experiments):
While I am not an expert in OOD detection and thus cannot assess the work based on its applied contributions, I am wondering if it is possible to have experiments where additional intuitions gained from the analysis of this work concretely improve the performance of some methods.

Author response:
While we agree that building this analysis into a framework that performs OOD detection and testing it on benchmark datasets is of value; this is a significant task which could stand on its own as a contribution to a more application focused venue, and so we consider it future work.

Reviewer comment (Writing):
There is some notational ambiguity and mathematical imprecision in the current version of the manuscript. Specific examples are provided in the questions section below.

Author response:
- Simplified the notation surrounding the loss gradient to improve the clarity of our equations

Reviewer comment (Writing):
It could help the readers better understand the setting if the authors clarify the meaning of "$f$ has been trained by a series of discrete steps composed from a sum of loss gradients ...". If this is just training via gradient descent, it might be easier to simply write down the GD equation.

Author response:
- Added a description to Theorem 3.1 indicating that these constraints are satisfied by gradient descent

Reviewer comment (Writing):
The input and output space of $f$ don't seem to be defined.

Author response:
- Added definitions for the input and output space of the network function $f$

Reviewer comment (Writing):
The notation $\frac{dL(f(x_i,\theta_s),y_i)}{d\hat{y}_{\theta_s(0)}}$ does not seem to be immediately interpretable. If my understanding of the meaning of this expression is correct, my suggestion is to define something like $L'(\cdot,\cdot)$ as the derivative of $L(\cdot,\cdot)$ with respect to its first argument, and use $L'(f(x_i,\theta_s(0)),y_i)$ instead.

Author response:
- Simplified the notation surrounding the loss gradient to improve the clarity of our equations