Paper: Double Momentum Method for Lower-Level Constrained Bilevel Optimization (ICLR-2024)

Reviewer comment (Theory):
Assumption 4 is also strange to me. More discussions are needed for this kind of boundedness assumption.

Author response:
**Q2: Assumption 4 is also strange to me. More discussions are needed for this kind of boundedness assumption.**

A2: For the hypergradient estimation $\nabla f_{\delta}(x,y)$, we have
>$$
|| \nabla f_ {\delta}(x,y)|| \\
    \leq || \nabla_x f(x,y)||+ \eta ||\nabla_ {xy}^ 2 g(x,y) || \cdot||\nabla\mathcal{P}_ {\mathcal{Y}\delta}(z)^ {\top}||\cdot ||\sum_ {i=0}^ {Q-1}\left((I_ {d_2}-\eta\nabla_ {yy}^2 g(x,y))\nabla\mathcal{P}_ {\mathcal{Y}\delta}(z)^{\top}\right)^i||\cdot||\nabla_yf(x,y)||\\
    =C_{fx}+\eta C_{gxy}C_{fy}\sum_ {i=0}^ {Q-1}||I_ {d_2}-\eta\nabla_ {yy}^ 2g(x,y)||^i\\
    \leq C_ {fx}+\dfrac{C_ {gxy}C_ {fy}}{\mu_ g} 
$$

Then, using the variance in Lemma 4, we can bound the norm of stochastic estimation, i.e.,  $||\bar{\nabla} f_{\delta}(x,y;\bar{\xi})||\leq C$. Therefore, we can easily obtain $||w||$ is bounded, since $w_{k+1}=(1-\alpha)w_{k}+\alpha\bar{\nabla} f_{\delta}(x_{k+1},y_{k+1};\bar{\xi}_{k+1})$ and $0<\alpha<1$. Similarly, we can bound $v$. Even if the norm of the gradient estimation is not bounded, we can also use the clipping method to make the norm of $v$ and $u$ bounded and finally make Assumption 4 hold.

Reviewer comment (Theory):
Why is the convergence rate only related to $d_2$ but not $d_1$?

Author response:
**Q3: Why is the convergence rate only related to $d_2$ but not $d_1$?**

A3: Since we use the zeroth order method to approximate the Jacobian matrix of the projection operator, the Lipschitz constant and variance of the approximation of the hypergradient is related to $d_2$. In addition, from our analysis, we can find that the convergence rate highly depends on these constants which make the convergence rate related to $d_2$. However, these constant does not affect these constants in the approximation of the hypergradient and therefore the convergence rate is not related to $d_1$.

Reviewer comment (Evaluation):
In the experiments, are the lower-level constraints active at the solution returned by the proposed algorithm?

Author response:
**Q4: Are the lower-level constraints active at the solution returned by the proposed algorithm?**

A4: Since the dimension of the lower-level variable is relatively large and the constraint parameter $r=1$ is relatively small,  using gradient update in $y$ will easily make $y$ get out of the constraint and lead to the projection on the boundary of the constraints. This makes the lower-level constraints active at the solution return by our method. We can easily check the constraint at the end of our algorithm.

Reviewer comment (Theory):
The proposed algorithm DMLCBO is based on double momentum technique. In previous works, e.g., SUSTAIN[1] and MRBO[2], double momentum technique improves the convergence rate to $\mathcal{\widetilde O}(\epsilon^{-3})$ while proposed algorithm only achieves the $\mathcal{\widetilde O}(\epsilon^{-4})$. The authors are encouraged to discuss the reason why DMLCBO does not achieve it and the theoretical technique difference between DMLCBO and above mentioned works.

Author response:
Your comments are greatly appreciated. SUSTAIN[1] and MRBO[2] using the Variance Reduction methods, i.e., STORM and SPIDER, both on the gradient estimation of $x$ and $y$.  These methods need stochastic gradient to be Lipschitz continuous. However, this condition is not satisfied in our method. Therefore, we can only use the traditional moving average method which leads to a different convergence rate. Similar results can be found (i.e., $\tilde{\mathcal{O}}(\epsilon^{-4})$) in [3,4] if the stochastic gradient is Lipschitz continuous.

Reviewer comment (Experiments):
In the experimental part, the author only shows the results of DMLCBO in early time, it will be more informative to provide results in the later steps.

Author response:
We highly value your feedback and comments. Please be aware that we've already included results in subsequent sections within our appendix. We extended the iteration count of our method to 100,000 to illustrate its convergence performance. Thank you for your consideration!

Reviewer comment (Evaluation):
In Table 3, DMLCBO exhibits higher variance compared with other baselines in MNIST datasets, the authors are encouraged to discuss more experimental details about it and explain the behind reason.

Author response:
Your comments are greatly appreciated. In our method, we use the zeroth order method which may have a large variance. This makes the poison data points found by our methods have high variance in different turns. And MNIST is a simple dataset and easily to be attacked which will increase the variance in the poison data points. Finally, these variances will highly affect the retrained model and lead to a large variance in the test accuracy.

Reviewer comment (Theory):
In order to achieve a stationary point with $\|| \nabla F (x) \|| \le \epsilon$, as outlined in Remark 2, the proposed algorithm necessitates a choice of the smooth parameter on the order of $O(\epsilon d_2^{-3/2})$. Consequently, the algorithm would require a minimum of approximately $\tilde{O}(d_2^8/\epsilon^8)$ iterations. It appears, however, that the authors aim to obscure this fact within their paper and retain the smooth parameter in their complexity result.

Author response:
Thank you very much for pointing out our mistake. We have modified the proof in the new version, hope you can check it again.

In our new proof of Theorem 1, we use the following new function,
>$$
\Phi_{k+1}\\
        =\mathbb{E}[F_{\delta}(x_{k+1})+\dfrac{10L_0^a c_l}{\tau\mu_g c_u}||y_{k+1}-y^*(x_{k+1})||^2 + c_l(\|w_{k+1}-\bar{\nabla} f_{\delta}(x_{k+1},y_{k+1})-R_{k+1}\|^2\\
        +\|\nabla_y g(x_{k+1},y_{k+1})-v_{k+1}\|^2)]
$$

where $0\leq a$ is a constant. We can obtain the following result. 
>$$
\dfrac{1}{K}\sum_{k=1}^ K\mathbb{E}[\dfrac{1}{2}\mathcal{M}_ k]
        \leq\dfrac{2m^{1/4}\sqrt{G}}{\sqrt{Kt}}+\dfrac{2\sqrt{G}}{(Kt)^{1/4}}+\dfrac{\delta L}{4\mu_g} C_ {gxy}(d_2+3)^{3/2}C_ {fy}(1+\dfrac{1}{\mu_g}(1-\eta\mu_g)).
$$

where $G=\dfrac{\Phi_1-\Phi^*}{\gamma c_l}+\dfrac{17t}{4K^2}(m+K)^{1/2}+\dfrac{4}{3tK^2}(m+K)^{3/2}+(m\sigma_{f}^2(d_2))t^2\ln (m+K)$. 
Then, setting $a=0$, we have $\sqrt{G}=\tilde{\mathcal{O}}(\sqrt{d_ 2})$.  Let $\delta=\mathcal{O}(\epsilon d_2^ {-3/2})$, $r$ randomly sampled from $\{0,1,\cdots,K\}$, we have $\mathbb{E}[\dfrac{1}{2}\mathcal{M}_ r]=\dfrac{1}{K}\sum_ {k=1}^ K\mathbb{E}[\dfrac{1}{2}\mathcal{M}_k]=\tilde{\mathcal{O}}(\dfrac{\sqrt{d_2}}{ K^{1/4}})\leq \epsilon$. Therefore, we have $K=\tilde{\mathcal{O}}(\dfrac{d_2^2}{ \epsilon^{4}})$. This is the first method with convergence analysis for LCBO with general constraints, which directly solves the BO problem. Comparatively, in [6], for the single-level nonsmooth problem, the authors derived an iteration number of $\mathcal{O}(d^{3/2}\delta^{-1}\epsilon^{-4})$. This outcome demonstrates that, even within the LCBO context, our method achieves results akin to those in the **single-level** problem setting."

Reviewer comment (Theory):
By Remark 2 on page 7, $\tilde{\mathcal{O}}(\frac{\sqrt{d_2}}{\delta K^{1/4}})\leq \epsilon$ implies that $K=\tilde{\mathcal{O}}(\frac{d_2^2}{\delta^4 \epsilon^4})$, NOT $\tilde{\mathcal{O}}(\frac{\sqrt{d_2}}{\delta \epsilon^4})$. Additionally, since $\delta=\mathcal{O}(\epsilon d_2^{-3/2})$ by (11), the iteration number $K=\tilde{\mathcal{O}}(\frac{d_2^8}{\epsilon^8})$.

Author response:
**Q1: By Remark 2 on page 7, the iteration number $K=\mathcal{O}(\dfrac{d_2^8}{\epsilon^8})$**.

A1: Thank you very much for pointing out our mistake. We have modified the proof in the new version, hope you can check it again.

In our new proof of Theorem 1, we use the following new function,
>$$
\Phi_{k+1}\\
        =\mathbb{E}[F_{\delta}(x_{k+1})+\dfrac{10L_0^a c_l}{\tau\mu_g c_u}||y_{k+1}-y^\*(x_{k+1})||^2 + c_l(\|w_{k+1}-\bar{\nabla} f_{\delta}(x_{k+1},y_{k+1})-R_{k+1}\|^2\\
        +\|\nabla_y g(x_{k+1},y_{k+1})-v_{k+1}\|^2)]
$$

where $0\leq a$ is a constant. We can obtain the following result. 
>$$
\dfrac{1}{K}\sum_{k=1}^ K\mathbb{E}[\dfrac{1}{2}\mathcal{M}_ k]
        \leq\dfrac{2m^{1/4}\sqrt{G}}{\sqrt{Kt}}+\dfrac{2\sqrt{G}}{(Kt)^{1/4}}+\dfrac{\delta L}{4\mu_g} C_ {gxy}(d_2+3)^{3/2}C_ {fy}(1+\dfrac{1}{\mu_g}(1-\eta\mu_g)).
$$

where $G=\dfrac{\Phi_1-\Phi^*}{\gamma c_l}+\dfrac{17t}{4K^2}(m+K)^{1/2}+\dfrac{4}{3tK^2}(m+K)^{3/2}+(m\sigma_{f}^2(d_2))t^2\ln (m+K)$. 
Then, setting $a=0$, we have $\sqrt{G}=\tilde{\mathcal{O}}(\sqrt{d_ 2})$.  Let $\delta=\mathcal{O}(\epsilon d_2^ {-3/2})$, $r$ randomly sampled from $\{0,1,\cdots,K\}$, we have $\mathbb{E}[\dfrac{1}{2}\mathcal{M}_ r]=\dfrac{1}{K}\sum_ {k=1}^ K\mathbb{E}[\dfrac{1}{2}\mathcal{M}_k]=\tilde{\mathcal{O}}(\dfrac{\sqrt{d_2}}{ K^{1/4}})\leq \epsilon$. Therefore, we have $K=\tilde{\mathcal{O}}(\dfrac{d_2^2}{ \epsilon^{4}})$. This is the first method with convergence analysis for LCBO with general constraints, which directly solves the BO problem. Comparatively, in [13], for the single-level nonsmooth problem, the authors derived an iteration number of $\mathcal{O}(d^{3/2}\delta^{-1}\epsilon^{-4})$. This outcome demonstrates that, even within the LCBO context, our method achieves results akin to those in the **single-level** problem setting."

Reviewer comment (Theory):
Since there is an additional loop to approximate the matrix inverse, it can be noted that the proposed algorithm DMLCBO is not fully single-loop.

Author response:
**Q4: Since there is an additional loop to approximate the matrix inverse, it can be noted that the proposed algorithm DMLCBO is not fully single-loop.**

A4: In bilevel optimization, the algorithm that alternately updates two variables is referred to as a single-loop algorithm even if it needs to approximate the matrix inverse, such as [1,2,3].

Reviewer comment (Theory):
Could you provide some representative class of problems that satisfy Assumption 3? Consider the simple example: $g(x,y)=(y-x)^2/2$, $\mathcal{Y}=[-1,1]$ and $\mathcal{X}=[-3,3]$. The projection operator $P_Y$ has a closed-form solution, but $\mathcal{P}_{\mathcal{Y}}(z^*)$ is not continuously differentiable in a neighborhood of $z^*$ when $x=1$ or $x=-1$.

Author response:
**Q2: Could you provide some representative class of problems that satisfy Assumption 3? Is Assumption 3 satisfied for all small values of $\eta$?**

A2: We acknowledge that the assumptions made in our study might have been overly stringent, potentially affecting the applicability of our research. The reason for employing these assumptions was to simplify the problem and facilitate theoretical analysis. However, we recognize that these strong assumptions may pose limitations in specific contexts.  We have modified Assumption 3 in our new version as follows,
>**Assumption 3.** a) If projection operator has a closed-form solution and **$z^\*=y^\*(x)-\eta \nabla_y g(x,y^\*(x))$ is not on the boundary of the constraint**, then $P_Y(z^*)$ is continuously differentiable in a neighborhood of $z^*$. In addition, in the neighborhood of $z^*$, $P_Y(z^*)$ has Lipschitz continuous gradient with constant $L$. **b) $y^\*(x)$ is continuously differentiable on a neighborhood of $x$**. 

In many complicated machine learning problems, the probability that $z^*$ falls exactly on the constraint boundary is very low. $z^*$ primarily reside either within or outside the boundary, i.e., a) the constraint is not active and $\nabla_y g(x,y^*)=0$; b) $y^*$ is on the boundary and $||\nabla_y g(x,y^*)||>0$. In these two cases,  the projection operator could be differentiable in the neighborhood of $z^*$, since we have the closed form of the projection operator and $z^*$ is not on the nonsmooth point. 



In addition, this assumption has been used in [4,5] to derive the hypergradient (6). The main contribution of this paper is not to use Assumption 3 to derive the hypergradient. The main purpose and contribution of our paper is to design a single-loop method with convergence analysis based on [4,5]. Therefore, even if the hypothesis is relatively strong, we believe that the proposal of our method is still meaningful. Specifically, we propose a single loop method for the LCBO and our method achieves convergence results akin to that in the **single-level** problem setting.

Reviewer comment (Theory):
What measures can be taken to verify that Assumption 4 is satisfied, or are there specific checkable sufficient conditions to ensure its validity?

Author response:
**Q3: What measures can be taken to verify that Assumption 4 is satisfied, or are there specific checkable sufficient conditions to ensure its validity?**

A3:  For the hypergradient estimation $\nabla f_{\delta}(x,y)$, we have

>$$
|| \nabla f_ {\delta}(x,y)|| \\
    \leq || \nabla_x f(x,y)||+ \eta ||\nabla_ {xy}^ 2 g(x,y) || \cdot||\nabla\mathcal{P}_ {\mathcal{Y}\delta}(z)^ {\top}||\cdot ||\sum_ {i=0}^ {Q-1}\left((I_ {d_2}-\eta\nabla_ {yy}^2 g(x,y))\nabla\mathcal{P}_ {\mathcal{Y}\delta}(z)^{\top}\right)^i||\cdot||\nabla_yf(x,y)||\\
    =C_{fx}+\eta C_{gxy}C_{fy}\sum_ {i=0}^ {Q-1}||I_ {d_2}-\eta\nabla_ {yy}^ 2g(x,y)||^i\\
    \leq C_ {fx}+\dfrac{C_ {gxy}C_ {fy}}{\mu_ g} 
$$

Then, using the variance in Lemma 4, we can bound the norm of stochastic estimation, i.e.,  $||\bar{\nabla} f_{\delta}(x,y;\bar{\xi})||\leq C$. Therefore, we can easily obtain $||w||$ is bounded, since $w_{k+1}=(1-\alpha)w_{k}+\alpha\bar{\nabla} f_{\delta}(x_{k+1},y_{k+1};\bar{\xi}_{k+1})$ and $0<\alpha<1$. Similarly, we can bound $v$. Even if the norm of the gradient estimation is not bounded, we can also use the clipping method to make the norm of $v$ and $u$ bounded and finally make Assumption 4 hold.

Reviewer comment (Theory):
On page 9, do the bilevel optimization problems related to training data poisoning attacks satisfy the smoothness and convexity assumptions? Note that “a network with two convolution layers and two fully-connected-layer layers for MNIST and a network with three convolution layers and three fully-connected-layer layers for Cifar10, where the Relu function is used in each layer.”

Author response:
**Q5: Do the bilevel optimization problems related to training data poisoning attacks satisfy the smoothness and convexity assumptions?**

A5: 
Thank you for bringing to our attention the issue in our paper. We acknowledge that our assumption regarding convexity and smoothness does not align with the lower-level problem in the training data poisoning attacks, particularly due to the use of DNNs with ReLU functions. Consequently, the theoretical analysis provided may not be directly applicable in this scenario. Nevertheless, it's essential to note that while our theoretical framework might not directly apply, it doesn't invalidate the capability of our algorithm to address this specific problem.

For other typos, we have corrected in our new version, and $\tilde{x}_ {k+1}=P_X(x_k-\dfrac{\gamma}{\sqrt{||w_{k}||}+G_0}w_k)$.