Paper: Newton Losses: Using Curvature Information for Learning with Differentiable Algorithms (ICLR-2024)

Reviewer comment (Evaluation):
Is there training curve of the real loss functions L during model training? Is there mismatch or conflicts between the MSE loss of the quadratic approximation and the real training loss for the real output y?

Author response:
> "Q1: Is there training curve of the real loss functions L during model training? Is there mismatch or conflicts between the MSE loss of the quadratic approximation and the real training loss for the real output y?"

The Newton Loss indeed has a different absolute loss value compared to the respective original loss, which as-per-construction of the Newton Loss does not pose a conflict. Further, we note that the original loss value is still available during training as the Newton Losses build upon the original loss.
Figure 3 shows the test accuracy during training and we have included axis labels in the revised PDF.

Reviewer comment (Experiments):
What is the training time for this new method, compared with original SGD method? Ex, the number of iterations and total training time. The validation test accuracy curve during training.

Author response:
> "Q2: what is the training time for this new method, compared with original SGD method? Ex, the number of iterations and total training time. the validation test accuracy curve during training."

Kindly see Supplementary Material B where we show runtimes for the experiments. In particular, it shows that the proposal has comparable runtime in most cases and only where the computation of the Hessian is slow, it can slow down training. Therefore, we also proposed training with the Fisher variant, which always has comparable runtime compared to the original SGD method.

Reviewer comment (Experiments):
There are very few analytical examples studied in the paper. This could help readers realize the importance of Fisher information matrix.

Author response:
It would be helpful if the reviewer could point us to what is meant with "analytical examples".
In particular, we would like kindly to point the reviewer to Supplementary Materials A, D, E and F, where we study the method from a theoretical perspective. For example, in SM D, we provide a variety of theoretical statements, and in SM F we illustrate analytically how the method would be applied to trivial losses.

Reviewer comment (Novelty):
There are general natural gradient methods in the literature, in particular the Wasserstein natural gradient method. This could be discussed in the introduction.

Author response:
We thank the reviewer for pointing us to these two references. Natural gradients are not directly applicable to differentiable algorithms (because they require particular structures of the loss, not given in the case of differentiable algorithm based losses). It is still very interesting to see that natural gradients have also been considered in combination with optimal transport, and extending these ideas to be applicable for differentiable algorithms could be interesting for future investigations. We have included them as suggested in the introduction.

Reviewer comment (Theory):
A suitable theoretical study is needed for the proposed paper. Some analytical examples can be presented to demonstrate the effectiveness of proposed method.

Author response:
It would be helpful if the reviewer could point us to what is meant with "analytical examples".
In particular, we would like kindly to point the reviewer to Supplementary Materials A, D, E and F, where we study the method from a theoretical perspective. For example, in SM D, we provide a variety of theoretical statements, and in SM F we illustrate analytically how the method would be applied to trivial losses.

Reviewer comment (Evaluation):
While the results show consistent improvements in the experiments, the improvement upon the best performing baselines seems relatively minor compared to the observed standard deviations.

Author response:
Indeed, our method leads to the greatest improvements for the earlier differentiable algorithms. We selected these two categories (sorting and shortest path) of differentiable algorithms explicitly because they each have a number of algorithms that improved upon each other to paint a full picture of the approach. Our motivation is indeed to provide the approach for other researchers to use for future differentiable algorithm setups, which are typically not as well optimized as, e.g., the recent Cauchy DSNs.

Reviewer comment (Experiments):
The method seems to especially improve performance on suboptimal baselines, which could transfer very well to potential new experimental setups.

Author response:
Indeed, our method leads to the greatest improvements for the earlier differentiable algorithms. We selected these two categories (sorting and shortest path) of differentiable algorithms explicitly because they each have a number of algorithms that improved upon each other to paint a full picture of the approach. Our motivation is indeed to provide the approach for other researchers to use for future differentiable algorithm setups, which are typically not as well optimized as, e.g., the recent Cauchy DSNs.

Reviewer comment (Evaluation):
The standard deviations should also be included in the main paper and not deferred to the appendix, as they seem very important for correctly interpreting the presented results.

Author response:
We would like to offer to include statistical significance tests for Tables 1, 2, and 3 for the camera-ready of the paper, considering the space restrictions. We believe that this would paint an even clearer picture than just including the raw standard deviations. Nevertheless, if you still believe that we should also include the standard deviations in exchange for a smaller table font size, we would accomodate that.

* We have now included the standard deviations in the experiment tables, and have indicated significant improvements visually.

Reviewer comment (Experiments):
In Tables 8-10 it looks like a wide range of the parameter $\lambda$ seems to be used in the experiments. Have the authors found these values purely empirically, or have they been chosen based on e.g. gradient norm? It would also be a useful addition to include an ablation study of how sensitive the performance is to this hyper-parameter for one of the experimental setups.

Author response:
We found these values empirically by considering the grid $\lambda\in[0.001, 0.01, 0.1, 1, 10, 100, 1000]$ for 1 seed (different seed from the seeds used for the tables).
We have started an ablation study to show the effects of $\lambda$ on the differentiable sorting experiment over the full grid. As these experiments will not finish within the rebuttal period, we offer to include the ablation study for the camera-ready.

* Further, the second seed (out of 10 seeds) for the $\lambda$ ablation study just finished.
Accordingly, we have now included a preliminary plot (Figure 4), which we will update for the camera-ready.
If space for the camera-ready permits, we offer to include one of the subplots in the paper.

Reviewer comment (Presentation):
In figure 3, does the plot show the number of training epochs on the x-Axis? It seems like this information is missing. This plot could also benefit from averaging over all seeds and including the standard deviations.

Author response:
Yes, it is the number of epochs. We added labels and extended the figure to include 95% confidence intervals.

Reviewer comment (Theory):
The main weakness of the presented approach is that the authors consider explicit construction of Hessian or Fischer matrices and directly inverse it. Matrix inversion is the numerically unstable operation in the ill-conditioned case, see (Higham, 1993 - https://eprints.maths.manchester.ac.uk/346/1/covered/MIMS_ep2006_167.pdf). This issue may be crucial in the single-precision format which is typically used in deep learning.

Author response:
We use Tikhonov regularization on the Hessian and Fisher matrices, which is a standard approach to improve their conditioning by "shifting the eigenvalues" (as the condition number of a matrix is the ratio of the largest to the smallest eigenvalue) (this is addressed in the paper from Equations (5) to (6)). Numerical problems are thus avoided. Note that the Fisher matrix, as an outer product, is necessarily positive semi-definite, and with Tikhonov regularization guaranteed to be positive definite.

Reviewer comment (Theory):
Storage of the full square matrix with a dimension equal to the number of model parameters can be prohibitive for large models. Thus, this step in the proposed method lacks detailed analysis and requires more in-depth investigation in terms of scalability and robustness to round-off errors.

Author response:
Indeed a "full square matrix with a dimension equal to the number of model parameters" could be prohibitively expensive.
However, in our work, we never consider such a matrix. Instead the Hessian or Fisher matrices are always of size $m\times m$ where **$m$ is the dimensionality of the output of the neural network** (see Section 3.1, all Hessian / Fisher matrices are wrt. $\mathbf{z}$, i.e., the output of the neural network).
To illustrate this, for a regular classification neural network, $m$ would correspond to the number of classes (often, 10--1000), which is always drastically smaller than the number of parameters (often millions to billions). In our paper, we consider sizes of $m$ in the range of 5 to 144.

We believe that many of the questions build on this misunderstanding.

Reviewer comment (Reproducibility):
Iterative methods for solving large symmetric linear systems can exploit Hessian-by-vector product operations without explicitly composing the Hessian matrix due to automatic differentiation tools.

Author response:
Indeed a "full square matrix with a dimension equal to the number of model parameters" could be prohibitively expensive.
However, in our work, we never consider such a matrix. Instead the Hessian or Fisher matrices are always of size $m\times m$ where **$m$ is the dimensionality of the output of the neural network** (see Section 3.1, all Hessian / Fisher matrices are wrt. $\mathbf{z}$, i.e., the output of the neural network).
To illustrate this, for a regular classification neural network, $m$ would correspond to the number of classes (often, 10--1000), which is always drastically smaller than the number of parameters (often millions to billions). In our paper, we consider sizes of $m$ in the range of 5 to 144.

We believe that many of the questions build on this misunderstanding.

Reviewer comment (Experiments):
How scalable is your approach in terms of the size of the dataset and the number of model parameters?

Author response:
The approach is scalable regardless of the number of model parameters and the size of the dataset. Scalability only depends on the number of outputs of the neural networks / number of inputs of the differentiable algorithm / loss. Another component can be the cost of computing the Hessian of a differentiable algorithm; however, for this reason, we also include the Fisher approach.

Reviewer comment (Theory):
What are specific issues related to the solving of weakly supervised learning problems? The authors mention non-convexity and exploding/vanishing gradients, but the same issues arise in the classical image classification models. Please provide more details regarding the mentioned learning problems' features.

Author response:
In this work, we deal with non-convexity and exploding/vanishing gradients specifically in the loss function (and not the neural network). In classical image classification, the loss is typically convex and does not have exploding/vanishing gradients (e.g., CrossEntropyLoss). Conceptually, we are dealing with problems where optimization of the loss function is the bottleneck of the learning problem and optimizing the neural network is easier than optimizing the loss. In image classification with convex losses, this is not the case.

Reviewer comment (Evaluation):
The presented runtime analysis in Supplementary B is confusing since in some cases the proposed method with additional costs per iteration is faster than the baseline. Please provide a more detailed description of the reason for such a result. Also, I disagree that the inversion of the matrix is inexpensive since it is $O(n^3)$ complexity and it is small only for the small dimension $n$. Therefore, we again come to question 1 on the scalability of the proposed method.

Author response:
Regarding the runtime analysis, the largest matrix that we need to invert is a well-conditioned 144x144 matrix, which is computationally very cheap (< 0.1 millisecond). Accordingly, in most experiments the runtime is only insignificantly increased, and as many factors affect the runtime (like GPU temperature, GPU die quality, CPU scheduling, PCIe interferences, etc.) the differences between runtimes are often smaller than the possible measurement precision of the runtimes. The only case where the runtime actually gets noticably more expensive was in the case of differentiable sorting with the Hessian because computing the Hessian of the differentiable sorting algorithms did not have closed-form solutions (using the Fisher resolves this concern.)

Reviewer comment (Evaluation):
The authors report only mean values of the target metrics in the main text. Could you please append the standard deviations, too, for a fair comparison of the methods?

Author response:
The standard deviations are reported in the Supplementary Material C. The sizes of the standard deviations are actually common in the field, which is why we followed the gold standard of using 10 seeds. We found that most improvements are actually statistically significant and we offer to indicate statistical significances in the camera-ready of the paper.

Reviewer comment (Presentation):
Captions to tables are typically placed above the table, not below.

Author response:
Thanks, we moved the captions on top of the tables.