Paper: Sobolev acceleration for neural networks (ICLR-2024)

Reviewer comment (Theory):
In both theorem 2 and 3 it is written that the convergence $\omega \to \omega^*$ is accelerated. How is this statement formally defined?

Author response:
In Theorem 2 and 3, $w\to w^\star$ is defined by $V(w) = \Vert w-w^\star \Vert_2^2 \to 0$ as $w$ follows the gradient flow $\dot{w} = -\nabla_w \mathcal{H}$. We apologize for the possible misleading.

Reviewer comment (Theory):
Related to the above question, the bound on the derivative of $V$ in theorem 2 and 3 only shows that Sobolev training is at least as good as L2 training, but it is not clear how to derive any concrete quantitative bound from this result.

Author response:
Regarding Theorem 2, there was a typo in our original statement. The correct version is:  
$\frac{dV}{dt} = -(w-w^*)^T \nabla_w \mathcal{H} < -(w-w^*)^T \nabla_w \mathcal{L} <0,$ demonstrating that Sobolev training ensures strictly faster convergence.

Reviewer comment (Reproducibility):
In the beginning of section 4 it is stated that $\omega^*$ and $e$ are sampled randomly, what distributions are they sampled from?

Author response:
For the sampling process, we generated $w^\star$ from the standard normal distribution and $e$ from the uniform distribution. We have updated our manuscript accordingly (refer to Section 4.1).

Reviewer comment (Theory):
Unfair/vacuous comparison: it is unfair to compare the convergence of gradient flow on $\mathcal{H}(w)$ and on $\mathcal{L}(w)$, since $\mathcal{H}(w)$ has extra terms that are added in an unnormalized way. To understand why this is unfair, consider another (trivial) way to accelerate the convergence speed of gradient flow, by rescaling the loss. Set $\tilde{\mathcal{L}}(w) = \mathcal{L}(w) + \mathcal{L}(w)$, then in the notation of Theorem 1, $\partial \tilde{V} / \partial t = -2 (w - w^*)^T \nabla_w \mathcal{L} \leq 2 \partial V / \partial t < 0$, which is already a significantly stronger 'acceleration' than Theorems 2 and 3 because it converges twice as fast (whereas Theorem 2 and 3 show only convergence that is not any slower than Theorem 1). Simply adding extra convex terms to the loss isn't sufficient to argue an acceleration phenomena, any more than rescaling the loss would be. It would be more fair to average the terms in the Sobolev loss, or to rescale so that $\mathcal{L}(0) = \mathcal{H}(0)$, but then Theorems 2 and 3 might not hold.

Author response:
We appreciate your time dedicated to reviewing our work and valuable comments.

We acknowledge that one may perceive the theoretical results as suggesting an unfair comparison. However, our primary objective in presenting the theorems is to highlight the convergence acceleration of Sobolev training in the $H^1$-norm compared to $L^2$-training. It's worth noting that the specific example $\tilde{\mathcal{L}}(w) = 2\mathcal{L}(w)$  you mentioned may seem to accelerate convergence, but this effect arises from the infinitesimal learning rate assumption and the student-teacher assumption in theoretical settings. Moreover, comparing $L^2$ and $H^1$ loss functions is a standard practice widely used in the literature [1,2].

In summary, our focus is on demonstrating the acceleration of $H^1$-training rather than suggesting a new loss function (such as $\tilde{\mathcal{L}}(w)$) for faster convergence. We hope you understand our intent.

Reviewer comment (Experiments):
Synthetic/toy experiments: the experimental evaluation in this work is entirely focused training toy objectives. While the Sobolev loss is shown to improve convergence on these objectives, it's not clear whether this benefit extends to actual datasets.

Author response:
In regards to experiments, we have revised our manuscript to include more practical experiments. Please refer to the updated manuscript and the official comment we have provided.