Paper: Three ways that non-differentiability affects neural network training (ICLR-2024)

Reviewer comment (Theory):
The authors imply that small VGG16 network weights are close to the "domain of non-differentiability." This is flawed, as weights and activations are not the same -- if for example the bias of the linear layer is a very large constant vector, then we have differentiability even from small weights.

Author response:
The image has been removed, and we have added a reference which shows that that ReLU networks do not have differentiable minima except in trivial cases.

Reviewer comment (Theory):
The authors generally frame their analysis as "practical networks do not satisfy theoretical ssumption (e.g., L-smooth loss functions), therefore they don't apply." This is a fundamental misunderstanding of the importance of theory. Theory tends to study a "toy" scenario, and derive results that are insightful even if they don't directly apply.

Author response:
A differentiable problem is not a good toy problem for a non-differentiable use case. That is the entire premise of our paper

Reviewer comment (Theory):
The authors imply that edge of stability training doesn't hold for non-smooth networks. Does this not contradict with the cited Cohen paper, which uses standard networks with ReLU nonlinearities?

Author response:
A couple of things in regard to the Cohen paper:

1) If anything, we agree with the conclusions made in that paper. In their discussions they mention that L-smoothness is not a good assumption to make when studying neural networks

2) Most of their paper  they use differentiable neural networks until the very end where they discuss ReLU networks under "experiments"

3) Note that in the gradient descent matching gradient flow argument, it held for all differentiable architectures but not some non-differentiable architectures. So there are differences in the non-differentiable case.

Reviewer comment (Theory):
Objectives are too simple and not representative of neural networks. The paper considers a one-layer linear model with an added ReLU activation at the end, which is an extremely simple version of neural networks. It further assumes that the labels $\mathbf{y}$ contain only non-positive entries, which is almost never the case in real neural network training. In particular, the two conditions combined enforces the neural network to almost never fit all training labels.

Author response:
1a. Our result are concerned with the structural properties of the NDGD sequence, and not with the specifics of the architecture, or how easy it is to train. 


1b. While simulations on the general non-differentiable non-convex case do show similar signs of non-capturing, it is hard to draw conclusions from those simulations because we do not have apriori expectations in what we expect the GD sequence to do in this setup.The non-negative labels are chosen in our analysis because with these labels, the loss function is convex, and therefore, we have a better handle on what we can expect from the GD sequence.

1c. We do not agree with the claim that single layer neural networks with ReLU activations are simple and/or not relevant. Single layer networks are precisely what are being trained in most transfer learning problems as the "regression head" or the "classification head"

Reviewer comment (Theory):
The argument that the GD dynamic differs on differentiable and non-differentiable objectives is not strong. In particular, the contradiction is constructed by running experiments on the simple objectives discussed above, and no rigorous of the whole claim is provided.

Author response:
2) We do not agree with this claim either. An example of one type of sequence not satisfying the properties of another type of sequence is rigorous proof of the fact that the two types of sequence do not have identical properties.

Reviewer comment (Theory):
The paper argues on Page 4 that ”it is consistently observed that the optimal parameter values ... non-differentiability for ReLU activations”. This argument is wrong since zeros of the parameters in the neural networks are not necessarily points at non-differentiability, which depends both on the neural network inputs and the bias.

Author response:
3) Thank you for pointing this out. We have removed the figure, and cited [1], who show that barring trivial cases, the local minima of. ReLU networks will be non-differentiable

Reviewer comment (Theory):
The argument in the last paragraph on Page 4 is also not strong since Tibshirani 2015 only proves the upper bound on the number of steps for subgradient methods. Saying the upper bound of one algorithm is different than the upper bound of the other algorithm is not sufficient to say that the two algorithms have different convergent behavior since the slower convergence rate might be an artifact of proof.

Author response:
4) Thank you for pointing this out. The bound is tight. We have updated the reference in our paper to [2] to reflect this fact

Reviewer comment (Novelty):
The conclusion in Section 4 of the paper does not significantly deviate from previous viewpoints. In summary, the paper shows that the subgradient gradient method does not give sparse solutions for LASSO, which is a known fact. The paper also does not theoretically extend this argument to more complicated objectives such as neural networks.

Author response:
5) We cite standard references in the field, highly cited papers, and concerns of unsure users on forums, all of whom expect the NDGD LASSO solutions to be sparse. Therefore the point is important to clarify. 

We have added results which shows that the paradoxical behavior seen with the penalized linear model is also seen with a penalized VGG16.

Reviewer comment (Theory):
In Section 5, the author argues that EoS have different behavior under non-differentiability, and uses Proposition 4 as a support. This is insufficient since the boundedness of the gradients is not related to either the progressive sharpening or the stabilization of the sharpness right above $2/\alpha$.

Author response:
6) Our claims are primarily about unstable convergence, and we use the defn. of stableness provided in definition 1 of [3]. As mentioned in the above paper, definition 1 is preferable to the definition using sharpness because the loss can still oscillate in the latter case. A similar defn. is provided in [4] -  "More specifically, Cohen et al. (2021) observe that when we run GD to train a neural network, the condition (1.2) fails, but contrary to the common wisdom from convex optimization, the training loss still (non-monotonically) decreases in the long run....We call this phenomenon “unstable” convergence".

Reviewer comment (Writing):
I am really confused about what the algorithm "NGD" is. The authors say it is "Non-differentiable gradient descents" without providing any concrete definition (how is $\tilde{\nabla}$ defined? Is it subgradient? If it depends on the problem, what is the concrete definition of it in Figure 1?). This makes it hard to evaluate this paper, as I do not know if it is the version of gradient applied to deep learning practice. Also, I would like to remind the authors that "NGD" is usually referred to as "Natural Gradient Descent", and I suggest the authors change the name.

Author response:
We have renamed the sequence to NDGD instead of NGD. We explain the symbols after equation 2 where we first mention it -- it is the gradient if the gradient at the point exist, and a heuristic measure otherwise.

Reviewer comment (Presentation):
The presentation of this paper needs to be polished. As an example, it is quite weird that the left subfigure of Figure 1 is not in a log scale, since there are only two or three points in the figure much above 0. Also, the subfigure even appears to be a screenshot with inconsistent color with the background. Also, on page 5, I do not think it is a proper way to use discussions in a forum to support claims.

Author response:
We have fixed the image. We believe that the claims in the forum are valid to cite because they have no bug in their code, and are raising a valid question given that multiple standard sources on the subject (cited in our paper) suggest that the NDGD solution ought to be sparse.

Reviewer comment (Theory):
Is Prop 1 correct? I do not check the reference, but at first glance, this appears to be strange, as intuitively it is possible that at $\theta_0$ the gradient is quite large and the iterations jump outside the unit ball.

Author response:
The result is correct. See page 15 of https://people.eecs.berkeley.edu/~wainwrig/ee227a/Bertsekas_1.2.pdf

Reviewer comment (Theory):
This paper extensively cites conclusions from other papers without any further explanation. For example,  
> Since the loss function described in equation (6) satisfies the Lipschitz condition (see equation (5)) with L = λ1, we have from Boyd et al. (2003) (Section 2 on the Constant Step Size) that...

Author response:
We have added a subsection on the subgradient method in the preliminaries, and explain why Boyd's notes are cited in multiple locations in our paper.

Reviewer comment (Theory):
Many connections are drawn based on imprecise observations/statements. For example, in the texts below,  
> we know from Figure 2 that the optimal weights lies close to points of non-differentiability  
How "close" is needed to make the Taylor expansion invalid?

Author response:
We realize that our histogram was causing confusions. We have removed it and added a reference to [1] who show that ReLU networks have non-differentiable minima except in trivial cases. We have changed the statement to "around points of non-differentiability", to remove imprecise terms like "close

Reviewer comment (Writing):
The first line under Eq. (3): y is a $N\times 1$ ...? (Typo)

Author response:
Both typos have been fixed. Thank you for pointing them out!

Reviewer comment (Writing):
Proof of Proposition 1: missing "." (Typo)

Author response:
Both typos have been fixed. Thank you for pointing them out!

Reviewer comment (Experiments):
Considering in the examples non-asymptotic and constant learning rate settings is distracting, as this means we cannot distinguish whether the asymptotic and diminishing learning rate settings would hold the same problems.

Author response:
2) We rewrote the preliminaries to address your question as to why the diminishing step-size is not being used. Essentially, we explain why every real world training is a constant step-size problem.

Reviewer comment (Writing):
In the abstract, "under-estimate" is ambiguous.

Author response:
7) Ambiguous terms have been removed.

Reviewer comment (Writing):
In the abstract, "unreliable" is also not clear.

Author response:
7) Ambiguous terms have been removed.

Reviewer comment (Theory):
In section 5, the result cited in the proof holds for decreasing learning rates.

Author response:
6) The result in section 5 holds with the constant learning rate. The last line on the constant step size in his notes: "We find that $f(x^} - f^{*}<=G^{2}\alpha ...".