Paper: Global Convergence Rate of Deep Equilibrium Models with General Activations (ICLR-2024)

Reviewer comment (Theory):
About the weight assumption. The authors assume that $W_{ij}\sim N(0,2\sigma_w^2/m)$ and $U\sim N(0,2/d)$. I do not understand the reason of using the scaling parameter "2". In ReLU case the scaling parameter "2" is commonly used for simplicity, but this paper investigates general activations.

Author response:
$\textbf{About the weight assumption: }$  Our purpose is to show that under the similar set of conditions as Ling et al. (2022), the gradient descent converges to a globally optimal solution at a linear convergence rate for the quadratic loss function for a DEQ with more general activation functions (linear or non-linear), not only with the ReLU (linear) as Ling at el. (2022). We use the same scale factors as Ling et al. (2022) to emphasize the $\textbf{similarity}$ in the conditions. In addition, by using the same conditions on $\mathbf{U}$ and $\mathbf{W}$ as Ling at el. (2022), we can use Lemma 4 in Ling at el. (2022) without any changes.

Reviewer comment (Theory):
About the existence and the uniqueness of $K$ (Proposition 12). In order to make sure Eq. (112) holds, one needs to make sure $2q^2\tilde{L}_q\sigma_w^2<1$ where 
$\tilde{L}_q=\frac{16L^2}{q^2} (\frac{\sigma_w^2}{m} \mathbb{E}G+\frac{3}{2})$
(as implied by Eq.(111)). However, Proposition 12 only requires that Assumptions 1 and 2 hold, i.e. $\sigma_w^2<\frac{1}{8L^2}$. I do not think this condition is sufficient to guarantee $2q^2\tilde{L}_q\sigma_w^2<1$.

Author response:
$\textbf{About the existence and the uniqueness of (Proposition 12):}$ We agree that there is a mistake here, and thank you for spotting this mistake. However, (112) still holds by assuming that $\sigma_w^2 < 1/(48 L^2)$ in Assumption 1 since we have
$$
2q^2 \tilde{L}\_q \sigma_w^2 =32 L^2 \sigma_w^2 \bigg(\frac{\sigma_w^2}{m}\mathbf{E}[G\_{11}]+\frac{3}{2}\bigg)=O(48 L^2 \sigma_w^2)
$$ as $m\to \infty$ (overparametrized DEQ)). 

At the first sight, we can tighten the the above assumption to $\sigma_w^2 <1/(24 L^2)$ by tightening the bound the Lipschitz constant $4L^2 \max(\alpha+1,\beta+1)^2/q^2$ for $\tilde{Q}_{\alpha,\beta}(\cdot)$ in Lemma 9 to $2L^2 \max(\alpha+1,\beta+1)^2/q^2$. We still think whether we can improve more in the revised version by tightening the above Lipschitz constant.

Reviewer comment (Theory):
The properties of $\mathbb{E}G_{11}$ are unclear. This makes the proof less rigorous.

Author response:
$\textbf{Moreover, the properties of $\mathbf{E}[G\_{11}]$ are unclear:}$ Sorry since we don't quite understand what you meant here. It seems to us that we forgot to state that we use the notation $\mathbf{G}:=\mathbf{G}(0)$, which leads to your concern. The matrix $\mathbf{G}:=(\mathbf{T}^*)^T \mathbf{T}^*$ where $\mathbf{T}^*$ is the equilibrium point of (2), which exists by using Lemma 2.  It is clear that $\mathbf{G}$ is a random matrix, and $G_{11}$ is a random variable, and $\mathbf{E}[G_{11}]$ is the expected value of this element.

Reviewer comment (Theory):
Lemma 10 (Proof of Lemma 4 in [1]) plays a key role in the proof. However, [1]'s proof works for ReLU function. The authors should explain the applicability of the proof to general activation functions.

Author response:
$\textbf{Lemma 10 (Proof of Lemma 4 in [1]) plays a key role in the proof:}$ The proof is exactly the same as the proof of Lemma 4 in [1], that is the reason why we remove the proof.  The idea is as follows. Observe that $\mathbf{G}_{ij}^{(l+1)}=(\mathbf{T}_i^{(l+1)})^T  \mathbf{T}_j^{(l+1)}$ where $\mathbf{T}_i^{(l+1)}$ is the $i$-th column of $\mathbf{T}$.  By using (1), we have
$$
\mathbf{T}_i^{(l+1)}=\varphi\big( [\mathbf{W} \mathbf{T}^{(l)}]\_i+[\mathbf{U}\mathbf{X}]\_i \big).
$$
By using the exact argument as the proof of Lemma 4 in [1], we can represent $[\mathbf{W} \mathbf{T}^{(l)}]\_i+[\mathbf{U}\mathbf{X}]\_i$ as $\mathbf{M}\mathbf{h}$ and $[\mathbf{W} \mathbf{T}^{(l)}]\_j+[\mathbf{U}\mathbf{X}]\_j$ as $\mathbf{M}\mathbf{h}'$.

As your concern, we will add this short idea to our revised version.

Reviewer comment (Theory):
Could you give examples of activation functions that would not fulfill the conditions of Theorem 8?

Author response:
Thank you very much for your feedbacks on our paper. For your question related to give examples of activation functions that would not fulfill the conditions of Theorem 8, our answer is as follows.  

Thank you for your interesting question. The Hermite expansion is known to exist for some class of functions such as the weighted Hilbert space $L^2(\mathbb{R}, e^{-x^2})$ since Hermite polynomials form an orthogonal basis for this Hilbert space. However, the coefficients of each orthogonal expansion depend on each function in $L^2(\mathbb{R}, e^{-x^2})$. As far as we know, the exact form of Hermite expansion is only given for some specific functions such as ReLU, tanh, or sigmoid. 

Essentially, the condition in our Theorem 8 requires that there are infinitely many positive coefficients in the Hermite expansion of that function. In (46), we give a sufficient condition for this condition to hold, however it is not easy to find a necessary condition or find a counter-example as your question.  

Looking forward to receive your better evaluation of our paper.

Reviewer comment (Novelty):
**contribution**: it is not clear what is the exact contribution of this work. To me it seems that it's merely an extension of the work of Ling et al. but it doesn't bring new theoretical ideas, proof techniques or closes a gap between theory and practice. This is largely reflected by the fact that a large portion of the text is extremely similar to the paper of Ling et al. It can also be seen in the fact that entire parts of the paper are dedicated to things that would usually be in the appendix like extended proofs, extended historical perspectives on generalization or examples. Moreover, while there is a claim that "a novel population Gram matrix" or "new form of dual activation with Hermite polynomial expansion" are introduced in this work, it is clear from reading them that they are direct extensions of Ling et al. or Daniely et al.

Author response:
$\textbf{About your comments about some text parts of Ling et al. paper appear in our work}$. We agree that that we use some texts in Ling et. al for the purpose of giving motivations why we need to design a novel Gram matrix $\mathbf{K}$ in for both linear (Ling et. al) and non-linear activations (our work), especially understanding why we need to bound the least eigenvalue of $\mathbf{G}(0)$ in Theorem 7 without reading Ling et. al paper to understand this. Based on your comments and avoid the using the texts from previous works, we will remove Lemma 2 and Lemma 3 and other repeated texts in our revised versions. We will only cite them in our revised version. 

Based on our contributions to develop theory for an important problem, we look forward to receiving your better evaluation. Thank you very much.

Reviewer comment (Novelty):
what are the contributions of this work on top of extending the proof of Ling et al. to other activation functions?

Author response:
$\textbf{What are the contributions of this work on top of extending the proof of Ling et al. to other activation functions?}$ As you can see, our work is to extend the work of Ling et. al to a more general setting. In Ling et. al work, the authors assume that the activation is the ReLU, which is a linear function. Our work extends it for more general activation functions which consist of non-linear activation functions such as sigmoid or tanh. To extend from a linear activation setting to a non-linear activation one, we need to extend some important concepts as follows:
 + Extend the dual activation concept in [Eq.11,Ling et. al] (or Daniely et. al 2016) to  a new class of dual activation functions in Definition 5. As you can see in [Eq.11,Ling et. al], for ReLU function, the dual activation function has a specific form, i.e., $Q(x)=\frac{\sqrt{1-x^2}+(\pi-arccos(x))x}{\pi}$. However, for the non-linear activation functions such as sigmoid or ReLU, it is very challenging to obtain an exact form for this dual activation function. Hence, our work proposes a method to overcome this difficulty by introducing a new concept called "a class of dual activation functions". 
 + Right before [Lemma 3, Ling et. al], the authors mention that by using the homogeneity (linear) of ReLU, the authors can use the definition of population Gram matrices in Definition 1 to obtain (12). For the non-linear activation functions that we consider in this work, we need to define a new type of population matrices as you can see in our Definition 6. To come up with this new type of population Gram matrices requires a lot of our efforts from mathematical viewpoints. It is not a straightforward extension of the population Gram matrices in Ling et. al.
+ In addition, the dual activation $Q(x)$ in [Lemma 3, Ling et. al] has a  Hermite expansion in [Lemma 7, Ling et. al] or [Daniely et. al 2016].  To deal with the non-linear activation functions, as mentioned above, we need to define a new class of dual activation functions. But, we need to show that the new class of dual activations also has a Hermite expansion. That is also a new contribution in this work.

Reviewer comment (Theory):
why is it important to extend the proof of Ling et al. to other activation functions?

Author response:
$\textbf{Why is it important to extend the proof of Ling et al. to other activation functions?}$. The Deep Equilibrium (DEQ) model was first introduced in [Bai et al.,2019]. This model has recently demonstrated impressive performance on a variety of large-scale vision and NLP tasks, often showing competitive performance relative to the state of the art [Bai et al., 2020].   In the above work, both linear and non-linear activation functions are usually used (eg. tanh is used in p.8,Bai et al.,2019). In Ling et. al, the authors only considered the ReLU DEQ. Our work contributes in understanding DEQ with non-linear activation functions. As we can see in practice, many facts hold for linear model but do not hold for non-linear ones. Hence, theoretical understanding of DEQ with non-linear activation functions are very important, that motivates our work.

Reviewer comment (Novelty):
The proof in this paper closely follows [Ling et al. (2023)](https://arxiv.org/pdf/2205.13814.pdf), and the modifications required for dealing with general activation functions appear very basic. The fundamental flow of the proof is essentially the same as in Ling et al., with direct correspondences such as Theorem 2 in Ling et al. to Theorem 7 here, and Theorem 3 in Ling et al. to Theorem 8 here. Although Hermite decomposition is introduced to handle cases beyond ReLU, much of the proof remains identical to the original. Given that ideas like Hermite decomposition are widely used in other literature (e.g., [Misiakiewicz (2022)](https://arxiv.org/abs/2204.10425)), the technical contribution seems incremental, raising concerns about the novelty and significance of the work.

Author response:
Thank you very much for your feedback. I agree that there are many methods to prove that $\mathbf{R}$ is positive definite. However, the key contribution here is not in how to prove Theorem 8 but in how you come up with such an $\mathbf{R}$ matrix or  $\mathbf{K}$ matrix for the non-linear activation DEQ. Please read my first answer to your question to explain why Ling et. al approach to design the matrix $\mathbf{K}$ does not work for the DEQ with non-linear activations. I copy it again here:

As you know, our work is to extend the work of Ling et. al to a more general setting. In Ling et. al work, the authors assume that the activation is the ReLU, which is a linear function. Our work extends it for more general activation functions which consist of non-linear activation functions such as sigmoid or tanh. To extend from a linear activation setting to a non-linear activation one, we need to extend some important concepts as follows:
  + Extend the dual activation concept in [Eq.11,Ling et. al] (or Daniely et. al 2016) to  a new class of dual activation functions in Definition 5. As you can see in [Eq.11,Ling et. al], for ReLU function, the dual activation function has a specific form, i.e., $Q(x)=\frac{\sqrt{1-x^2}+(\pi-arccos(x))x}{\pi}$. However, for the non-linear activation functions such as sigmoid or ReLU, it is very challenging to obtain an exact form for this dual activation function. Hence, our work proposes a method to overcome this difficulty by introducing a new concept called "a class of dual activation functions". 
  + Right before [Lemma 3, Ling et. al], the authors mention that by using the homogeneity (linear) of ReLU, the authors can use the definition of population Gram matrices in Definition 1 to obtain (12). For the non-linear activation functions that we consider in this work, we need to define a new type of population matrices as you can see in our Definition 6. To come up with this new type of population Gram matrices requires a lot of our efforts from mathematical viewpoints. It is not a straightforward extension of the population Gram matrices in Ling et. al.
   + In addition, the dual activation $Q(x)$ in [Lemma 3, Ling et. al] has a  Hermite expansion in [Lemma 7, Ling et. al] or [Daniely et. al 2016].  To deal with the non-linear activation functions, as mentioned above, we need to define a new class of dual activation functions. But, we need to show that the new class of dual activations also has a Hermite expansion. That is also a new contribution in this work.

Reviewer comment (Novelty):
The literature review appears to lack depth. Several works on the application of NTK to DEQ, which are covered in detail by [Ling et al. (2023)](https://arxiv.org/pdf/2205.13814.pdf), are overlooked, with only Ling et al. being mentioned. The Introduction starts with a general discussion on deep learning but omits a review of directly relevant literature. A suggested improvement would be to introduce DEQ first, then present existing analyses and their challenges, providing a more comprehensive and relevant overview within a similar page length.

Author response:
2. $\textbf{Could you provide more detailed information about the relevant literature?}$ Thank you very much for your introducing to some papers of which we were not aware such as Misiakiewicz (2022). We will add more relevant papers in our revised version. If you have any other paper suggestions, please let us know. 

Based on our feedback, we look forward to receiving your better evaluation.

Reviewer comment (Theory):
The statement in the abstract, "any general activation that has bounded first and second derivatives," requires modification. In Theorem 8, the authors assume non-vanishing Taylor coefficients on the dual activation function, and bounded first and second derivatives do not necessarily suffice to meet this assumption.

Author response:
Thank you very much for your question. As we mention before (253), it holds that $\mathbf{E}[G_{ii}]$ does not depend on $i$. This means that $\mathbf{E}[G_{ii}]=\mathbf{E}[G_{jj}]$ for $i\neq j$. On the other hand, from (17), we have
$$
Q_{ij}(\nu_{ij}):=\tilde{Q}\_{\sqrt{2(\frac{\sigma_w^2}{m}\mathbf{E}[G_{ii}]+1)},\sqrt{2(\frac{\sigma_w^2}{m}\mathbf{E}[G_{jj}]+1)}  }(\nu\_{ij})= \tilde{Q}_{\alpha,\alpha}(\nu\_{ij}),
$$ where
$$
\alpha:=\sqrt{2\bigg(\frac{\sigma\_w^2}{m}\mathbf{E}[G\_{ii}]+1\bigg)}.
$$
In Theorem 8, we assume that $\tilde{Q}\_{\alpha,\alpha}(\nu\_{ij})=\sum\_{r=0}^{\infty}\mu\_{r,\alpha}^2\nu\_{ij}^r$ because $Q\_{ij}(\nu\_{ij})= \tilde{Q}\_{\alpha,\alpha}(\nu\_{ij})$ by the above analysis.

Reviewer comment (Theory):
It might be helpful if the paper illustrated the challenges and difficulties of dealing with general activation functions by contrasting the proof methods with those from previous literature, which could highlight aspects that might have been overlooked.

Author response:
1. $\textbf{It might be helpful if the paper could illustrate the challenges and difficulties of dealing with general activation functions by contrasting the proof methods with those from previous literature, which could highlight aspects that I might have overlooked.}$

As you know, our work is to extend the work of Ling et. al to a more general setting. In Ling et. al work, the authors assume that the activation is the ReLU, which is a linear function. Our work extends it for more general activation functions which consist of non-linear activation functions such as sigmoid or tanh. To extend from a linear activation setting to a non-linear activation one, we need to extend some important concepts as follows:
  + Extend the dual activation concept in [Eq.11,Ling et. al] (or Daniely et. al 2016) to  a new class of dual activation functions in Definition 5. As you can see in [Eq.11,Ling et. al], for ReLU function, the dual activation function has a specific form, i.e., $Q(x)=\frac{\sqrt{1-x^2}+(\pi-arccos(x))x}{\pi}$. However, for the non-linear activation functions such as sigmoid or ReLU, it is very challenging to obtain an exact form for this dual activation function. Hence, our work proposes a method to overcome this difficulty by introducing a new concept called "a class of dual activation functions". 
  + Right before [Lemma 3, Ling et. al], the authors mention that by using the homogeneity (linear) of ReLU, the authors can use the definition of population Gram matrices in Definition 1 to obtain (12). For the non-linear activation functions that we consider in this work, we need to define a new type of population matrices as you can see in our Definition 6. To come up with this new type of population Gram matrices requires a lot of our efforts from mathematical viewpoints. It is not a straightforward extension of the population Gram matrices in Ling et. al.
   + In addition, the dual activation $Q(x)$ in [Lemma 3, Ling et. al] has a  Hermite expansion in [Lemma 7, Ling et. al] or [Daniely et. al 2016].  To deal with the non-linear activation functions, as mentioned above, we need to define a new class of dual activation functions. But, we need to show that the new class of dual activations also has a Hermite expansion. That is also a new contribution in this work.

Reviewer comment (Novelty):
Could you provide more detailed information about the relevant literature?

Author response:
2. $\textbf{Could you provide more detailed information about the relevant literature?}$ Thank you very much for your introducing to some papers of which we were not aware such as Misiakiewicz (2022). We will add more relevant papers in our revised version. If you have any other paper suggestions, please let us know. 

Based on our feedback, we look forward to receiving your better evaluation.