Paper: What does automatic differentiation compute for neural networks? (ICLR-2024)

Reviewer comment (Novelty):
Theorem 1 is a minor extension of Theorem 3.6 in Lee et al. (2023).

Author response:
**Theorem 1 is a minor extension of Theorem 3.6 in Lee et al. (2023).**

We believe our Theorem 1 is more than a minor extension of Theorem 3.6 in (Lee et al., 2023). First, our theorem considers a larger class of neural networks than the previous theorem: the latter restricts a network to have no residual connections (page 9 of (Lee et al., 2023)), while the former does not have this restriction (page 4 of our paper). Second, for the same network, our Theorem 1 proves the same conclusion given in Theorem 3.6 of (Lee et al., 2023), but under a weaker assumption. The latter theorem states that if proxy gradients are in the Bouligand subdifferential, then AD computes an element of the Clarke subdifferential. Our Theorem 1 extends this result as follows: the same conclusion holds even if we use a wider class of proxy gradients (namely those in the Clarke subdifferential). We believe this extension is an important addition to a line of recent works (e.g., [Bertoin+, Boursier+]) on understanding the effects of the choice of proxy gradients. We have clarified this point in the revised version (page 5).

In addition, we would like to emphasize that our Theorem 6 further generalizes Theorem 3.6 in (Lee et al., 2023) by considering an even larger class of neural networks. The latter theorem considers networks that do not contain usual residual connections, convolutional layers, and normalization layers such as BatchNorm (pages 4 and 9 of (Lee et al., 2023)) and do not allow minibatched inputs; hence, this prior result is not applicable to most convolutional networks used in practice. In contrast, our Theorem 6 allows a minibatch setup and a network with general residual connections, convolutional layers, and normalization layers (page 4 of our paper); thus, this result is applicable to a wider range of neural networks including practically-used convolutional networks. 

[Bertoin+] Numerical influence of ReLU'(0) on backpropagation, NeurIPS, 2021.\
[Boursier+] Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputs, NeurIPS, 2022.

Reviewer comment (Novelty):
Lemma 2 and Lemma 3 are complicated formulations of essentially known facts.

Author response:
**Lemmas 2 and 3 are complicated formulations of essentially known facts. Why not referring to the fact that this is a known issue?**  

We believe Lemmas 2 and 3 are not just a mere reformulation of known facts. As the reviewer pointed out, we indeed proved these two lemmas motivated by the known fact: the Clarke subdifferential is generally “incompatible with addition”, e.g., as in the example given by Kakade and Lee (2018). However, this incompatibility does not always guarantee the incorrectness of AD. For example, we showed that AD always outputs an element of the Clarke subdifferential under some conditions (Theorems 1, 4, 5, 6, and 8). Given this, what we essentially proved in Lemmas 2 and 3 is the following: under certain conditions, the incompatibility issue can actually show up in some neural networks. In this regard, Lemmas 2 and 3 are not an immediate consequence of the known incompatibility issues.

Following the reviewer’s suggestion, we have added the following clarification to the revised version (page 6): “Lemmas 2 and 3 are based on the incompatibility of the Clarke subdifferential with addition (Clarke et al., 1998; Kakada and Lee, 2018).”

Reviewer comment (Theory):
Lemma 3 and Theorem 4 are contradictory.

Author response:
**Lemma 3 and Theorem 4 are contradictory.**

Lemma 3 and Theorem 4 (or Theorem 6) do not contradict each other. Theorem 4 shows that AD computes an element of the Clarke subdifferential of a network (satisfying Condition 1 with distinct bias parameters) if the following condition holds: the proxy gradient $D^{AD} \rho_l(x)$ of $\rho_l$ (i.e., the activation function in the $l$-th layer) can be written as $\lambda_l D^-\rho_l(x) + (1-\lambda_l) D^+\rho_l(x)$, where $\lambda_l \in [0,1]$ should be *fixed across all $x \in\text{ndf}(\rho_l)$ and the layer $l$*. That is, the proxy gradient should be a convex combination of the left- and right-hand derivatives, where the same combination weight $\lambda_l$ is used for all inputs $x$ (within the $l$-th layer).

On the other hand, our counterexample in Lemma 3 uses $\rho_l =\text{HardSigmoid}$ with $D^{AD} \rho_l(-3) = D^{AD} \rho_l(3) = 0$. That is, $D^{AD} \rho_l(x)$ uses the left-hand derivative at $x=-3$ and the right-hand derivative at $x=3$. Due to this, there *cannot exist* a combination weight $\lambda_l \in [0,1]$ such that $D^{AD} \rho_l (x) = \lambda_l D^-\rho_l(x) + (1-\lambda_l) D^+\rho_l(x)$ for all $x \in\text{ndf}(\rho_l)$. Hence, this counterexample does not satisfy the condition in Theorem 4, and thus it is not contradictory to Theorem 4. We have clarified this point in the revised version (page 6).

Reviewer comment (Theory):
Conditions 1 and 2 are very complicated without an intuitive explanation of what they mean.

Author response:
**Conditions 1 and 2 are very complicated without an intuitive explanation of what they mean.**

Condition 1 states that (i) each analytic pre-activation function $\tau_l$ should have no bias parameters at all, or have bias parameters that should not be “shared” across multiple neurons in the same layer (e.g., as in fully-connected layers), and (ii) each activation function $\rho_l$ should apply the same piecewise-analytic function elementwise. Similarly, Condition 2 states that (i) each analytic pre-activation function $\tau_l$ should have no bias parameters at all, or have bias parameters that can be shared across multiple neurons in the same layer (e.g., as in convolutional layers), and (ii) each activation function $\sigma_l$ should be the composition of a maxpool function and a elementwise piecewise-analytic function. To improve readability, we have added this explanation to the revised version (page 4).

Reviewer comment (Theory):
Theorem 8 lacks discussion on the rate of positive outcomes and complexity.

Author response:
**Theorem 8 lacks discussion on the rate of positive outcomes and complexity.**

**(1) When does the algorithm stop with non-empty $\mathcal P_l$?**

One condition that guarantees $\mathcal P_l$ to be non-empty is the condition in Theorem 6: when a network satisfies Condition 2 with shared bias parameters and only trivial maxpools. For networks without shared bias parameters or with non-trivial maxpools, we could not (and we expect to be challenging to) find a generic sufficient condition that guarantees the non-emptiness of $\mathcal P_l$, which in turn implies the correctness of AD. This is why we proposed Theorem 8 (with Algorithm 1) in the first place, and why we empirically checked $\mathcal P_l \neq \emptyset$ using experiments.

**(2) Does it occur often? Does it always occur?**

As stated in Section 4 of the initial draft, we empirically observed that $\mathcal P_l$ was always non-empty during the training of the following networks: fully-connected networks with various activation functions (ReLU6, HardTanh, HardSigmoid) on the MNIST dataset, and various convolutional networks (ResNet18, VGG11, VGG11-BN) on the CIFAR-10 dataset. Together with Theorem 8, this implies that AD always computed an element of the Clarke subdifferential in our experiments.

Reviewer comment (Novelty):
Lemma 2 and Lemma 3 — the example in Lemma 2 is explicitly mentioned in Kakade and Lee 2018 and the mechanism in Lemma 3 is exactly the same, namely "incompatibility with addition". A similar comment holds for Lemma 7. Is there anything else beyond these lemmas? Why not refer to the fact that this is a known issue?

Author response:
**Lemmas 2 and 3 — the example in Lemma 2 is explicitly mentioned in Kakade and Lee 2018 and the mechanism in Lemma 3 is exactly the same, namely "incompatibility with addition". A similar comment holds for Lemma 7. Is there anything else beyond these lemmas? Why not refer to the fact that this is a known issue?**  

We believe Lemmas 2 and 3 are not just a mere reformulation of known facts. As the reviewer pointed out, we indeed proved these two lemmas motivated by the known fact: the Clarke subdifferential is generally “incompatible with addition”, e.g., as in the example given by Kakade and Lee (2018). However, this incompatibility does not always guarantee the incorrectness of AD. For example, we showed that AD always outputs an element of the Clarke subdifferential under some conditions (Theorems 1, 4, 5, 6, and 8). Given this, what we essentially proved in Lemmas 2 and 3 is the following: under certain conditions, the incompatibility issue can actually show up in some neural networks. In this regard, Lemmas 2 and 3 are not an immediate consequence of the known incompatibility issues.

Following the reviewer’s suggestion, we have added the following clarification to the revised version (page 6): “Lemmas 2 and 3 are based on the incompatibility of the Clarke subdifferential with addition (Clarke et al., 1998; Kakada and Lee, 2018).”

Reviewer comment (Theory):
Lemma 3 and Theorem 4 are contradictory. D- and D+ are never explicitly defined, so I assume that for a piecewise analytic function it corresponds to the derivative of the function on the left and on the right at a piece change point. In this case, the Bouligand derivative is {D-, D+} and the Clarke is its convex hull. So in Theorem 4, choosing all lambda = 0, one is in the setting of Lemma 3. I suspect that Theorem 4 is false for this reason. Is my reasoning correct?

Author response:
**D- and D+ are never explicitly defined.** 

We defined $D^-$ and $D^+$ in Section 2.1 of the initial draft; they denote the left- and right-hand derivatives as the reviewer expected. To improve readability, we have added a pointer to these definitions right after Theorem 4 in the revised draft.

**Lemma 3 and Theorem 4 are contradictory.**

Lemma 3 and Theorem 4 (or Theorem 6) do not contradict each other. Theorem 4 shows that AD computes an element of the Clarke subdifferential of a network (satisfying Condition 1 with distinct bias parameters) if the following condition holds: the proxy gradient $D^{AD} \rho_l(x)$ of $\rho_l$ (i.e., the activation function in the $l$-th layer) can be written as $\lambda_l D^-\rho_l(x) + (1-\lambda_l) D^+\rho_l(x)$, where $\lambda_l \in [0,1]$ should be *fixed across all $x \in\text{ndf}(\rho_l)$ and the layer $l$*. That is, the proxy gradient should be a convex combination of the left- and right-hand derivatives, where the same combination weight $\lambda_l$ is used for all inputs $x$ (within the $l$-th layer).

On the other hand, our counterexample in Lemma 3 uses $\rho_l =\text{HardSigmoid}$ with $D^{AD} \rho_l(-3) = D^{AD} \rho_l(3) = 0$. That is, $D^{AD} \rho_l(x)$ uses the left-hand derivative at $x=-3$ and the right-hand derivative at $x=3$. Due to this, there *cannot exist* a combination weight $\lambda_l \in [0,1]$ such that $D^{AD} \rho_l (x) = \lambda_l D^-\rho_l(x) + (1-\lambda_l) D^+\rho_l(x)$ for all $x \in\text{ndf}(\rho_l)$. Hence, this counterexample does not satisfy the condition in Theorem 4, and thus it is not contradictory to Theorem 4. We have clarified this point in the revised version (page 6).

Reviewer comment (Theory):
I really have troubles understanding the difference between Condition 1 and Condition 2 with "trivial max-pool". This is very complicated and I believe there should be a qualitative description of what these conditions mean. For example, I cannot tell: is one of the conditions more general than the other? Similarly: is there a difference between Theorem 4 and Theorem 6? Lastly, due to the same concern as above, I suspect that Theorem 6 is false.

Author response:
**Any difference between Condition 1 and Condition 2 with trivial maxpools? Any difference between Theorem 4 and Theorem 6?**

Condition 2 with only trivial maxpools is a strict generalization of Condition 1: (i) the former includes the latter as a special case, and (ii) the former allows shared bias parameters while the latter allows only distinct bias parameters. (i) can be observed by setting $M_l=N_l=C_l$ and $A_{l,c}=e_c{\bf 1}_B^\top$. (ii) follow directly from the definition of the two conditions. We stated these observations before and after Condition 2 in the initial draft. We have made this difference clearer in the revised draft (page 4).

Since Condition 2 with only trivial maxpools is a strict generalization of Condition 1, Theorem 6 is a strict generalization of Theorem 4 that allows shared bias parameters (which are commonly used in convolutional layers and normalization layers).

Reviewer comment (Theory):
Something that I didn't understand regarding the notion of distinct bias parameters - is it ever an issue that some bias parameters might be equal during training?

Author response:
We thank the reviewer for raising this question. In our paper, “distinct” bias parameters essentially means that the bias parameters used in different neurons should be all distinct, as variables not as values. It is fine for these bias parameters to take the same values as long as they are distinct as variables.

For instance, consider two pre-activation functions $\tau : \mathbb{R}^2 \times \mathbb{R}^1 \to \mathbb{R}^2$ and $\tau’ : \mathbb{R}^2 \times \mathbb{R}^2 \to \mathbb{R}^2$ defined by $\tau(x_1, x_2, b_1) = (x_1+b_1, x_2+b_1)$ and $\tau’(x_1, x_2, b_1, b_2) = (x_1+b_1, x_2+b_2)$. Then, $\tau$ does not have distinct bias parameters since the two outputs of $\tau$ use the same bias parameter $b_1$. In contrast, $\tau’$ has distinct bias parameters since all of its outputs use distinct bias parameters. Here, the values of $b_1$ and $b_2$ are not important and they do not change the fact that $\tau’$ has distinct bias parameters.

Reviewer comment (Theory):
The results in the theorems are not intuitive and are difficult to follow.

Author response:
**The results in the theorems are not intuitive and are difficult to follow.**

In the initial draft, we gave an informal overview of our results in Section 1 (pages 2-3), and provided a more detailed yet still high-level description of our results right after each result (pages 5-8). For example, Theorem 1 is informally introduced in Section 1: “Theorem 1 shows that AD always computes an element of the Clarke subdifferential, if the minibatch size is one and all non-differentiable neurons have distinct bias parameters.” The theorem is explained in more detail yet still informally in Section 3.1: “Theorem 1 states that if a network $\Psi$ has distinct bias parameters and the minibatch size is one, then AD computes an element of the Clarke (or Bouligand) subdifferential of $\Psi$ as long as the proxy gradient $D^{AD} \rho_l(x)$ is an element of the Clarke (or Bouligand) subdifferential for all $l$ and $x$.” We hope these informal and/or high-level explanations would make readers easier to follow our results.
In the revised draft, we have further improved readability by adding more explanations on our problem setup (Conditions 1 and 2) and more discussions on our theoretical results (pages 4–6).

Reviewer comment (Theory):
It is not clear how the numerical or theoretical result given by gradient descent differs from that given by subgradient descent.

Author response:
**It is not clear how the numerical or theoretical result given by gradient descent differs from that given by subgradient descent.**

We do not compare gradient descent and subgradient descent algorithms. The main objective of our paper is to understand the output of AD when neural networks contain non-differentiable activation functions such as ReLU and maxpools. Our theoretical results aim to characterize sufficient conditions under which AD always computes an element of the Clarke subdifferential (Theorems 1, 4, 5, 6, and 8), and other conditions under which AD might not do so (Lemma 2, 3, and 7). Our numerical results aim to verify whether our sufficient conditions are satisfied or not when we train neural networks via minibatch stochastic gradient descent (SGD), where the “gradients” used in SGD are computed by AD. We empirically observed that our sufficient conditions were always satisfied, i.e., AD always computed an element of the Clarke subdifferential in our experiments.

Reviewer comment (Reproducibility):
Are the results in Table 2 given by gradient descent, subgradient descent, or Clarke subgradient descent?

Author response:
**Are the results in Table 2 given by gradient descent, subgradient descent, or Clarke subgradient descent?**

As we noted in Section 4, all the results in Table 2 are given by minibatch stochastic gradient descent (SGD) algorithm implemented in PyTorch, where the “gradients” used in SGD are computed by AD (which is standard in practice). Hence, this learning algorithm is in general not a gradient, subgradient, or Clarke subgradient descent algorithm, since AD might not compute the true gradient or a (Clarke) subgradient for general neural networks (e.g., Lemmas 2 and 3). However, in our experiments, we verified that our sufficient conditions for AD to compute an element of the Clarke subdifferential were always satisfied. This result implies that the networks in Table 2 were in fact trained by a Clarke subgradient descent algorithm.