Paper: On Penalty Methods for Nonconvex Bilevel Optimization and First-Order Stochastic Approximation (ICLR-2024)

Reviewer comment (Theory):
Prop 3.1. How to get Prop. 3.1 from Thm. E.3? How does the additional ${\rm Span}(\,...\,, \nabla_y f(x,y^*))$ appear?

Author response:
*Q1. Prop 3.1. How to get Prop. 3.1 from Thm. E.3?:* We can plug the Theorem E.3 with $w=(x,\sigma)$ and $\theta = y$, and $func(w,\theta) := h_{\sigma}(x,y) = \sigma f(x,y) + g(x,y)$. The image of $\nabla_{w\theta}^2 func$ is the span of columns, which is $span(\nabla_{xy}^2 h_{\sigma}(x,y), \nabla_y f(x,y))$.

Reviewer comment (Theory):
Thm. 3.2. Can we derive the explicit form of $\nabla \psi(x)$? Or we can only define it implicitly as the limit of $\nabla \psi_{\sigma}(x)$ when ${\sigma \rightarrow 0^+}$?

Author response:
*Q2. Can we derive the explicit form of $\nabla \psi$:* Our result is also the explicit form of $\nabla \psi$, but yes, you can also directly derive the formula $\nabla \psi$ following the same steps in our proof in Appendix E.5. In that case, we would need the Lipschitz continuity of solution sets in $x$ at $T(x,0^+)$.

Reviewer comment (Theory):
Appendix E.6. Proof of Thm. 3.6. After "Then, since we have Assumption 1, we get", why we have $2 (g(x,y_{\sigma}^*) - g(x,z_p^*)) \ge \mu \Vert y_{\sigma}^* - z_p^* \Vert^2$? It seems like the QG condition in Karimi et al. (2016), but it seems like this reference only shows that it can be implied by the PL condition in the unconstrained case. But how can we get this in the constrained case?

Author response:
*Q3. Appendix E.6. Proof of Thm. 3.6:* Here we used Assumption 1, (proximal-EB): by definition of the proximal operation, $$g(x,y) \ge g(x,prox(y)) + \frac{1}{\rho} \|y - prox(y)\|^2 \ge g(x,z) + \frac{1}{\rho} \|y - prox(y)\|^2.$$ 
Then use Assumption 1 to lower bound $\frac{1}{\rho} \|y - prox(y)\|^2$. We will add this step in our proof.

Reviewer comment (Theory):
Proof of Thm. E.3 in Appendix. Why do we always have $\Omega(\delta) v = \nabla^2_{\theta w} f(w,\theta^\ast) {\rm d}w$? What if $\nabla_{\theta w}^2 f(w,\theta^\ast)$ is zero, then the RHS is also zero?

Author response:
*Q4. Proof of Thm. E.3 in Appendix:* We took $dw$ in the range space of $\nabla^2 f$, so that $\nabla^2 f dw$ is always nonzero. If $\nabla^2 f = 0$, then the span of the image is $\{\emptyset\}$, so the proposition is trivial. We will add a comment that we do not consider the case when $\nabla^2 f = 0$.

Reviewer comment (Theory):
Appendix B.2. Can the authors explain more about the necessity of using the Moreau envelope? What is the benefit of using the Moreau smoothing? What will happen if it is not used? The reviewer does not fully understand the technical challenge 1. Why does it fail if one simply estimates the value of $(\lambda^*,\nu^*,y^*)$ for a given $x$ and plugs into the expression of $\nabla \psi_{\sigma}(x)$ in Thm. 3.2?

Author response:
*Q5. Appendix B.2. Why Moreau envelope?* Thank you for the question, this is a great point. Yes, we can always solve the inner problems with sufficiently good accuracy first, before updating the outer variable. Though with stochastic noises and constraints, it is not entirely obvious to obtain (near) optimal multipliers along with solutions -- essentially, this is another constrained optimization in stochastic settings, which is a complicated subproblem by itself. Using Moreau Envelop is a convenient technique to side-step such issues, and further it becomes easy to design the algorithm in a single-loop. Nevertheless, as the reviewer questioned, we also think that it is an interesting question whether we can directly work with multipliers without Projection steps and/or Moreau Envelop smoothing.

Reviewer comment (Theory):
Assumption 1: Was this assumption considered in previous work? Moreover, the quantifiers are unclear: "satisfies (4) for all $y$ that satisfies... with some positive $\mu,\delta,\sigma_0$". Do you mean that there exist $\mu,\delta,\sigma_0$ so that $y$ satisfies... *implies* (4)? This should be properly revised.

Author response:
*Assumption 1:* The most well-studied assumption is the strong-convexity of $g$, which implies Assumption 1. EB is essentially PL in the constrained settings, and considered only in a few recent work (Shen and Chen, 2023). Thank you for the suggestion, we will revise the part accordingly.

Reviewer comment (Theory):
Assumption 3 vs. 4: It is assumed that $f,g\to\infty$ as $\|y\|\to\infty$, but then that $\|y\|\leq O(1)$. Why aren't these in contradiction to one another?

Author response:
*Assumption 3-4:* Assumption 3 is about the functions, and Assumption 4 is about the domain of $y$, and they do not contradict each other.

Reviewer comment (Writing):
1st paragraph after (P): "continuously differentiable *and* smooth" – are these synonymous?

Author response:
*continuously differentiable and smooth:* A function can be continuously differentiable but not smooth; an example is $f(x) = sign(x) \cdot x^2$.

Reviewer comment (Writing):
Theorem 1.1 (Informal): the phrase "at least one sufficiently regular solution path exists" is unclear at this point. What is even a solution path (not previously mentioned or defined)?

Author response:
*Theorem 1.1 (Informal):* We will clarify that the detailed meaning of our phrases is explained over Section 3, and the formal version is presented in Theorem 3.6.

Reviewer comment (Writing):
Title: What do you mean by "first-order stochastic approximation"?

Author response:
*"first-order stochastic approximation" in title:* We design a stochastic algorithm for solving Bilevel optimization with only first-order derivatives of objective functions, hence first order stochastic approximation.

Reviewer comment (Writing):
Page 2: The third paragraph states that there are results under the uniform PL condition, which seemed to imply that there is a method for the PL condition, but the next paragraph claims that there is no algorithm finding the stationary point under the PL condition. Could you clarify this?

Author response:
*The third paragraph states that there are results under the uniform PL condition ... Could you clarify this?:* Yes, there are previous works that study the PL condition for the Bilevel optimization, but all existing works use some relaxed notion of ($\epsilon$-approximate) stationarity rather than the most straight-forward stationarity notion of $\|\nabla \psi(x)\| \le \epsilon$. For example, we can alternatively use the $\epsilon$-stationarity of penalty formulation, or a $\epsilon$-KKT point for $P_{con}$ as we discussed in Remark 3.7. Our contribution is to establish strong bond between such alternative stationarity measures and the original stationarity measure.

Reviewer comment (Theory):
Page 3: It is implied that Assumption 1 guarantees the local Lipschitz continuity of solution sets (Assumption 5), but it seems that it is not explicitly stated (with proof) anywhere.

Author response:
*It is implied that Assumption 1 guarantees ... not explicitly stated (with proof) anywhere:* This implication is stated in our Lemma 3.5 and Lemma D.2.

Reviewer comment (Theory):
Page 3: PL and proximal EB conditions are almost identical, and you only assume the proximal EB within the neighborhood of solutions. Then, how were you able to get an improved rate over existing rate for the PL condition? Most of the explanations of the algorithm are deferred to the appendix, and it was not clear for what reason that the proposed algorithm is more efficient even with weaker condition. Could you comment on this?

Author response:
*PL and proximal EB conditions are almost identical ... the proposed algorithm is more efficient even with weaker condition. Could you comment on this?:* Thank you for your question. To our best knowledge, we are not aware the specific rates discussed in stochastic settings with PL condition. In the deterministic setting, (Chen et al., 2023) gave a $O(\epsilon^{-3})$ guarantee with PL condition for the penalty method, so it matches to what we provide. Our result should be viewed as an approach generalizing to a broader class of instances, rather than improving the compleixty.

Reviewer comment (Theory):
Page 9: Can Theorem 3.6 be proven with Assumption 5 rather than Assumption 1?

Author response:
*Can Theorem 3.6 be proven with Assumption 5 rather than Assumption 1?:* Good point. The answer is yes, you can show the same result (with replacing $\mu$ with something else), as long as we assume that Assumption 5 holds globally for all $x$ and $\sigma$ (and yes, Assumption 1 is slightly stronger than that).

Reviewer comment (Theory):
Finally you introduce a relaxation of the constraint in P_pen. Intuitively, we understand that for sigma small enough, this relaxation should recover P_con and this is what you formalize by introducing appropriate conditions. So far, this is clear. What I find less clear is the sentence “Although finding the solution of (Ppen) is a more familiar task than solving (P)”. For small σ, from Theorem 1.1., there must be some conservation of difficulty (at least for some σ). It would be helpful to have some more intuition on that (one or two sentences). I.e. it seems miraculous to end up with a formulation that suddenly becomes so much easier to solve.

Author response:
*Page 1/2 - Clarification on "a familiar task":* The reason that we described the penalty formulation as a slightly easier task is that it is essentially a single-level optimization (although technically, we are still dealing with the "min-max" structure due to $\min_z g(x,z)$. So, as you pointed out, this is still non-trivial). We have edited this paragraph to make the point clear in the revision.

Reviewer comment (Theory):
The result of Theorem 1 is clearly interesting but it could be expected that a smaller value of σ would enforce more weight on the penalty and hence ultimately recover the solution to (1).

Author response:
*Page 3 - Theorem 1.1:* You are right, while Theorem 1.1 makes an intuitive sense, our main contribution is the explicit quantification of the intuition: we exactly state the dependence of the bias induced by using the penalty formula.

Reviewer comment (Writing):
In section 3, we don’t really understand what you are doing.

Author response:
*Page 5 - What Section 3 Intro doing:* We are trying to explain why and when the penalty formulation is not a good approximation of the original problem. Here we also introduce some non-standard notations and concepts, as our approach starts from converting the landscape approximation question to the question of whether differentiation with respect to $x$ and $\sigma$ is commutable.

Reviewer comment (Presentation):
I think you can spare some space by removing the definition of Lipschitz continuity from the main body of the paper (this is a relatively well-known concept and it would free some space for some other (perhaps more crucial) explanations).

Author response:
*Page 5 - Lipschitz continuity:* While  Lipschitz continuity is a widely known concept for real-valued functions, we do not think it is the case for set-valued mappings.

Reviewer comment (Writing):
The whole introduction in section 3 is confusing. It is not really clear what you are trying to do until the last sentences. I.e. we understand you want to show examples in which ∇ψ(x) ≠ lim_{σ→0} ∇ψ_σ(x). But it is not completely clear why, especially given that you seem to already have introduced in Assumption 1 a condition that ensures that ∇ψ_σ can be made arbitrarily close to ∇ψ.

Author response:
*Page 5 - What Section 3 Intro doing:* We are trying to explain why and when the penalty formulation is not a good approximation of the original problem. Here we also introduce some non-standard notations and concepts, as our approach starts from converting the landscape approximation question to the question of whether differentiation with respect to $x$ and $\sigma$ is commutable.

Reviewer comment (Presentation):
Before introducing Examples 1 and 2, I would add a sentence insisting on the fact that in both examples, the non-existence of ∇ψ comes from the discontinuity in ψ. Although you don’t have much space, it would help to have a plot of the solution sets for both examples (or perhaps briefly mention the solution sets for both cases).

Author response:
*Page 6/7 - Section 3.1:* To clarify the derivation of the assumptions in Section 3.1, we moved two examples from the beginning of Section 3 to Section 3.1. Furthermore, we added the following sentence at the beginning of Section 3.  

> The following two examples illustrate the obstacles encountered when claiming $\lim_{\sigma \rightarrow 0} \nabla \psi_{\sigma}(x) \rightarrow \nabla \psi(x)$ or even when simply ensuring the existence of $\nabla \psi(x).$

Reviewer comment (Presentation):
The central part of section 3, i.e. the part that connects this section to Theorem 1.1., is Theorem 3.2. This theorem should appear way earlier or at least you should clearly explain that you derive the conditions appearing in Theorem 1.1. I would rewrite section 3 as follows: start by clearly indicating that you will derive conditions for the second relation in Theorem 1.1. to hold (i.e. the fact that ∇ψ_σ(x) is an O(σ) approximation of ∇ψ).

Author response:
*Page 6/7 - Theorem 3.1 (new numeration, previously Theorem 3.4):* We thank the reviewer for the comments. We also wanted to state Theorem 3.1 as early as possible. However, we believe that providing appropriate explanations for the assumptions is equally important. That's why we included examples and assumptions before stating the theorem. We appreciate the reviewer's suggestion on the organization. Following the suggestion, we added the following paragraph to the introduction of Section 3 before presenting the examples.  

> In what follows, we derive two assumptions: one concerning solution-set continuity (Assumption 5) and another addressing the regularities of lower-level constraints (Assumption 6). Under these assumptions, we establish the main theorem of this section, Theorem 3.2. This, in turn, leads to the derivation of the second inequality in our landscape analysis result, as presented in Theorem 1.1 and as shown in Theorem 3.6.

Reviewer comment (Writing):
I would rewrite section 3.1. by giving more intuition on how you derive Assumption 6. Just a couple sentences of explanation. (I think readability of the paper would benefit from, for example, removing the definition of Lipschitz continuity and replacing it with a more elaborate explanation of the derivation of Assumption 6).

Author response:
*Page 6/7 - Section 3.1:* To clarify the derivation of the assumptions in Section 3.1, we moved two examples from the beginning of Section 3 to Section 3.1. Furthermore, we added the following sentence at the beginning of Section 3.  

> The following two examples illustrate the obstacles encountered when claiming $\lim_{\sigma \rightarrow 0} \nabla \psi_{\sigma}(x) \rightarrow \nabla \psi(x)$ or even when simply ensuring the existence of $\nabla \psi(x).$

Reviewer comment (Theory):
I would remove Proposition 3.1. or I would make it simpler. The proposition is not clear and neither is its connection to the rest of the paper or its explanation (i.e. “Geometrically speaking, perturbations in …”).

Author response:
*Page 7 - Section 3.2:* We included Proposition 3.4 (new numeration, previously Proposition 3.1) in the main text since we believe that it is important to highlight the consequences of Assumptions 5 and 6. To clarify this point, we rearrange Section 3.2 as follows. We explain our main results (Theorem 3.1) and the remarks first. Then, we present Proposition 3.4. We believe that this gives an idea of the proof of Theorem 3.1.

Reviewer comment (Presentation):
I find the algorithm developed in this paper to be both interesting and important. It seems that the landscape analysis section is quite lengthy. Would it be possible to condense the content and incorporate the first-order algorithm into the main text?

Author response:
In addition, we would like to thank you for your interest in our algorithm. Regarding the comment W1, we also believe that both landscape analysis and algorithms are equally important, but we choose to leave only the landscape analysis in the main text. Please see the general comments to all reviewers for further details.

Reviewer comment (Writing):
On page 7, Definition 7: Does $\lambda_{\mathcal{I}}$ implicitely depend on $y$ in Equation (8)? What is the domain of $\mathcal{L}_{\mathcal{I}}$?

Author response:
*(3):* We appreciate your comments regarding the domain in Definition 7. To clarify this point, we define the Lagrangian restricted to a set of constraints first in Definition 7. The domain of $\mathcal{L}_I$ is added here.
> Definition 7. $\cdots$ We also define the Lagrangian $L_I (\cdot | x,\sigma): \mathbb{R}_+^{|\mathcal{I}|} \times \mathbb{R}^{m_2} \times \mathbb{R}^{d_y} \rightarrow \mathbb{R}$ restricted to a set of constraints $\mathcal{I} \subseteq [m_1]$ $\cdots$

In addition, we added the following sentence at the end of Section 3.1. We will update the manuscript accordingly to clarify this point.
> When the context is clear, we always let $\mathcal{I}$ in (8) be $\mathcal{I}(y)$ at a given $y$.

Reviewer comment (Theory):
On page 9, Assumption 8: Does Assumption 8 implicitly assume that the active set does not change?

Author response:
*(5):* Assumption 8 is about the minimum singular value of the LICQ condition in Assumption 6 and fairly standard in constrained optimization. This is about whether there are linearly dependent active constraints (so there could be multiple $\lambda^*$) at the solution $y^*$, and we do not want that to avoid further complication.

Reviewer comment (Theory):
On page 30, Proof of Theorem E.5: It appears that in the last step of the proof, $g(x,y) - g^*(x)$ is bounded by $O(\sigma^2)$. Would this observation be useful for improving the convergence rate?

Author response:
*(12):* Thank you for pointing this out. We corrected the proof of Theorem E.5 as follows:
>\begin{align*}
        g(x,y) - g(x,z) = g(x,y) - g^*(x) \le \sigma^2 l_{f,0} L_T + \frac{\sigma^2 L_T^2}{2\rho} = O(\sigma^2),
\end{align*}