Paper: Near-Optimal Solutions of Constrained Learning Problems (ICLR-2024)

Reviewer comment (Theory):
In line 4 of Algorithm 1, the **exact** minimizer of this unconstrained **nonconvex** optimization problem cannot be obtained. This minimization error should be considered which I envision will involve the parameterization error $\nu$ plus the error rate of the corresponding convex unconstrained optimization. In addition, for the stochastic version of Algorithm 1, the error of $\widetilde{\ell}_i\approx \ell_i$ should also be considered in line 4, not just in the dual update step in line 5.

Author response:
**(1.1)** The effect of solving line 4 approximately is quite mild, as considered in Theorem 2 of [7]. Essentially, it introduces a bias on the supergradient used to update $\lambda$ in line 5. While this bias limits the precision at which the $\lambda_p^\star$ can be recovered, its effects do not accumulate over the trajectory. Indeed, suppose we replace line~4 with a $\rho-$approximate oracle, i.e., one capable of finding $f\_{\theta}^{\dagger} \in \mathcal{F}\_{\theta}$, such that

$$
L(f\_{\theta}^{\dagger}, \lambda_p) \leq \min\_{\theta\in\Theta} L(f\_{\theta}, \lambda_p) + \rho 
$$
    
for $\rho \geq 0$. Then, $\ell(f_{\theta}^{\dagger})$ is an  approximate supergradient of $g_p$, i.e.,
    
$$
g_p(\lambda) \geq g_p(\lambda_p) + (\lambda - \lambda_p)^T \ell(f_{\theta}^{\dagger}) - \rho
$$

Hence, Lemma 4.1 holds with an extra term depending on $\rho$, namely,
$$
\lim_{t \to \infty} g_p^{\text{best}}(t | \lambda(t_0)) \geq D^{\star}_p - \left(\frac{\eta S^2}{2}+\rho\right) \quad \text{a.s.,}
$$

which propagates to Proposition 4.2. We will extend our results to account for a~$\rho-$approximate oracle in the revised manuscript.
    
It is worth noting that, even in the non-convex case, there is a myriad of results showing that near-optimal solutions (small $\rho$) can be obtained for line 4 using gradient descent, e.g., for overparametrized neural networks [3,4,5,6]. Since this approximation affects the results in a controlled manner, we hope that this addresses the reviewer's concern.
    
**(1.2)** This issue is related to the estimation error (i) that we mention in our conclusion as potential future work as well as in our overall response above when describing the context of our paper. As we explain there, the focus of this paper is on analyzing the algorithmic error (iii) that arises from retrieving a specific model from Alg. 1 rather randomizing over its iterates. We then argue that considering other sources of error (such as the estimation error) is beyond the scope of this paper considering that (a) the approximation and estimation errors were already analyzed in detail in [7] (Section 3) and (b) analysis of the algorithmic error already involves considerable challenges and novel proof techniques (as noted by the reviewer as well as Y13X and rUm3). Naturally, we welcome further discussions with the reviewer if they believe the paper contributions are not sufficient.
    
That being said, we provide a potential roadmap (following [7], Prop 3.4) extending our results in order to show that estimation error, though non-trivial, is not impossible.

[7] tackled this issue in the context of the duality gap between an empirical version of $(\text{D}_p)$ and $(\text{P}_p)$ using uniform convergence. Indeed, many results from learning theory (such as those based on VC dimension or Rademacher complexity) establish that, with probability $1-\delta$ over sample draws of $N$ samples,

$$
\left|\mathbb{E}\_{\mathfrak{D}}\left[\ell_i\left(f\_{\theta}(x), y\right)\right]-\frac{1}{N} \sum_{n=1}^N \ell_i\left(f\_{\theta}\left(x_n\right), y_n\right)\right| \leq \zeta_i(N, \delta),
$$

for all $\theta \in \Theta$ [18]. The function $\zeta_i(N, \delta)$ establishes the sample complexity of the problem. Under uniform convergence, every optimization problem can be solved within $\zeta_i(N, \delta)$ (with high probability). The implications of this were characterized in [7] Theorem 1, where they proved a near-feasible primal solution exists, **but not that it can be recovered** without randomization as we do here (see [7] Theorem 3).

Reviewer comment (Theory):
The error of $f_{\theta}(t)$ as well as $\widetilde{\ell} _ i\approx \ell _ i$ will also cause the approximation error of $g_p(\lambda(t))\approx\widetilde{\ell} _ 0[f _ {\theta}(t)]+\lambda(t)^{\top}\widetilde{\ell}[f _ {\theta}(t)]$ for selecting the best prima iterate. This should also be considered.

Author response:
**(1.2)** This issue is related to the estimation error (i) that we mention in our conclusion as potential future work as well as in our overall response above when describing the context of our paper. As we explain there, the focus of this paper is on analyzing the algorithmic error (iii) that arises from retrieving a specific model from Alg. 1 rather randomizing over its iterates. We then argue that considering other sources of error (such as the estimation error) is beyond the scope of this paper considering that (a) the approximation and estimation errors were already analyzed in detail in [7] (Section 3) and (b) analysis of the algorithmic error already involves considerable challenges and novel proof techniques (as noted by the reviewer as well as Y13X and rUm3). Naturally, we welcome further discussions with the reviewer if they believe the paper contributions are not sufficient.
    
That being said, we provide a potential roadmap (following [7], Prop 3.4) extending our results in order to show that estimation error, though non-trivial, is not impossible.

[7] tackled this issue in the context of the duality gap between an empirical version of $(\text{D}_p)$ and $(\text{P}_p)$ using uniform convergence. Indeed, many results from learning theory (such as those based on VC dimension or Rademacher complexity) establish that, with probability $1-\delta$ over sample draws of $N$ samples,

$$
\left|\mathbb{E}\_{\mathfrak{D}}\left[\ell_i\left(f\_{\theta}(x), y\right)\right]-\frac{1}{N} \sum_{n=1}^N \ell_i\left(f\_{\theta}\left(x_n\right), y_n\right)\right| \leq \zeta_i(N, \delta),
$$

for all $\theta \in \Theta$ [18]. The function $\zeta_i(N, \delta)$ establishes the sample complexity of the problem. Under uniform convergence, every optimization problem can be solved within $\zeta_i(N, \delta)$ (with high probability). The implications of this were characterized in [7] Theorem 1, where they proved a near-feasible primal solution exists, **but not that it can be recovered** without randomization as we do here (see [7] Theorem 3).

Reviewer comment (Theory):
You could also obtain the difference of $\ell_0$ between parameterized and convex problems in both Theorem 3.5 and Proposition 4.2 so that the solution of the unparameterized problem is not only nearly feasible but also nearly optimal.

Author response:
**(1.3)** As we mention in the general response, the near-feasibility results for $f\_{\theta}(\lambda^{\star}_p)$ derived in this work can be combined with the duality gap bound from [7], [15] to bound the near-optimality $|\ell_0(f\_{\theta}(\lambda^{\star}_p)) - P^{\star}|$. Namely, as long as $P^{\star}_p - D^{\star}_p \leq \Delta$, it holds that

\begin{equation}
\tag{1}
| P_p^\star - \ell_0(f\_{\theta}(\lambda^{\star}_p))| \leq \\| \lambda^{\star}_p \\|_1 \\| \ell(\phi^{\star})  - \ell(f\_{\theta}(\lambda^{\star}_p)) \\| \_{\infty} + (1 + \|\lambda^{\star}_p \|_1 ) M\nu + \Delta 
\end{equation} 

Taking $\Delta$ from [7] Proposition 3.3, and the bound in our Corollary 3.9 yields the desired result. Note that, plugging these values, (1) is of the same order as Corollary 3.9, i.e., no new terms appear, only the constants are different. We will include this bound as a corollary in the revised version of the manuscript. For the reviewer's reference, we provide a sketch of the derivations below.

Using the fact that $P^{\star}_p - D^{\star}_p \leq \Delta$ and $P^{\star}_u \leq P^{\star}_p$, we obtain that

$$
P_u^\star - D_p^\star \leq \Delta \Leftrightarrow 
\ell_0(\phi^\star) - \ell_0(f\_{\theta}(\lambda^\star_p)) \leq \lambda^{{^\star}T}_p \ell(f\_{\theta}(\lambda^\star_p)) + \Delta
$$

Since $\phi^*$ is feasible [$\ell(\phi^\star)) \leq 0$] and $\lambda^\star_p \succeq 0$ we obtain

$$
\begin{aligned}
\ell_0(\phi^\star) - \ell_0(f\_{\theta}(\lambda^\star_p)) & \leq \lambda^{{\star}T}_p  (\ell(f\_{\theta}(\lambda^{\star}_p) -\ell(\phi^{\star})) + \Delta  \\\ 
& \leq \Vert \lambda^\star_p \Vert_1  \Vert \ell(\phi^{\star}) - \ell(f\_{\theta}(\lambda^{\star}_p)) \Vert\_{\infty} +  \Delta
\end{aligned}
$$

To conclude the derivation of (1), we use $\nu$-universality and $M$-Lipschitz continuity (Assumptions 3.1 and 3.3) to get that there exists $\theta^\prime$ such that $\vert \ell_i(\phi^{\star}) - \ell_i(f\_{\theta^\prime})\vert \leq M\nu$. The duality gap bound then implies the approximate saddle-point relation

$$
L(f\_{\theta}(\lambda^\star_p), \lambda^\star_u) - \Delta \leq L(f\_{\theta}(\lambda^\star_p), \lambda^\star_p) \leq L(f_{\theta^\prime}, \lambda^\star_p) + \Delta,
$$

whose second inequality gives

$$
\begin{aligned}
\ell_0(\phi^\star) - \ell_0(f\_{\theta}(\lambda^\star_p)) &\geq \ell_0(f\_{\theta^\prime}) - M\nu - \ell_0(f\_{\theta}(\lambda^\star_p))
\newline
& \geq \lambda^{{\star}T}_p (\ell(f\_{\theta}(\lambda^\star_p)) - \ell(\phi^\star)) - \Delta - (1 + \Vert \lambda^{\star}_p\Vert_1 )M\nu
\newline
& \geq - \Vert \lambda^{\star}_p \Vert_1 \Vert \ell(\phi^{\star}) - \ell(f\_{\theta}(\lambda^{\star}_p)) \Vert\_{\infty} - (1 + \Vert \lambda^{\star}_p\Vert_1 ) M\nu - \Delta
\end{aligned}
$$

which completes the proof.

Reviewer comment (Theory):
You could obtain finite time convergence result, i.e. when replacing $\lambda^{\rm best}$ in Proposition 4.2 with $\lambda _ T^{\rm best}$, the best iterate up to a fixed iteration $T$, what is the convergence rate involving $T$? I think this is possible by using the existing convergence rates on the convex constrained optimization problems in terms of the constraint violation (like $\sum _ {i=1}^m \alpha _ i\max(\ell _ i, 0)$ for some constants $\alpha _ i>0$) as well as optimality. Then we upper bound difference of the convergence rates between the convex and parameterized versions.

Author response:
**(1.4)** Indeed, finite time convergence can be obtained because the maximization of the dual function is a concave maximization problem. Note that the dual function $g_p(\lambda)$ is concave, irrespective of whether problem $(P_p)$ is convex, so the procedure described in Algorithm 1 to maximize $g_p$ is a standard projected supergradient ascent algorithm, well-studied in the literature [12], [19].
    
Projected dual supergradient ascent algorithms have a quadratic rate of convergence. That is, when using a diminishing step size, for $\lambda_T$ to be $\delta$-optimal,  $O(1/\delta^2)$ iterations are needed ([12] Section 2.4, [20]).
    
Despite the fact that when using a diminishing step-size, $\lambda^{\text{best}}_t$ gets arbitrarily close to $\lambda^{\star}_p$, and, $f\_{\theta}(\lambda(t))$ gets arbitrarily close to **the set** of Lagrangian minimizers $\mathcal{F}(\lambda^{\star}_p)$, the difficulty lies in establishing which element of that set the iterates are converging towards. This is what we establish in this work. Although Section 4 already translates Theorem 3.5 into an algorithimc result, we have not fully explored the dynamics of Algorithm 1, which is a promising topic of future work.

Reviewer comment (Novelty):
About the example of Counterfactual Fairness Constraints — Is this example proposed by you? If not, cite the sources of the model and data. Otherwise, say that this example is proposed by you.

Author response:
**(1.1)** Counterfactual fairness was not defined by us. The definition was proposed and studied in [17] (see Definition 5). We use a constrained learning formulation to tackle that problem using the COMPAS dataset [21] as in [7], [15], [22].

Reviewer comment (Writing):
About the example of Counterfactual Fairness Constraints — What do the constraints mean in this example? You could add an explanation to the paper.

Author response:
**(1.2)** In this setting, each constraint represents a requirement that the output of the classifier be near-invariant to changes in the protected features (in our case gender and race). For instance, the output of the classifier should be (almost) the same whether, for all other variables kept the same, the gender of a sample is changed from male to female (and vice-versa) or whether the race is changed from caucasian to african-american, latino to caucasian...

Reviewer comment (Theory):
About the example of Counterfactual Fairness Constraints — Should we use $\mathbb{E} _ {x,y}$ in both objective and constraints, with the same distribution of $(x,y)$?

Author response:
**(1.3)** While our formulation does not require objective and constraints to be defined over the same distribution, this is how the requirement of counterfactual fairness is formulated (see Definition 5 in [17]).

Reviewer comment (Writing):
About the example of Counterfactual Fairness Constraints — Why are some $x$ bolded and some are not? Are they the same $x$?

Author response:
**(1.4-1.5)** The lack of boldface in the $x$ of the constraints of the learning problem are a typo. Effectively, all $x$ in the problem refer to the feature vector $\mathbf{x}$. As for the input of the function, in retrospect we must agree with the reviewer that our abuse of notation causes confusion. In an attempt to highlight the protected features, we collect them into the separate vector $\mathbf{z}$. In other words, $\mathbf{x} = [\bar{\mathbf{x}}; \mathbf{z}]$, so that we can write $f\_{\theta}(\mathbf{x}) = f\_{\theta}(\bar{\mathbf{x}}, \mathbf{z})$. This is more in keep with the notation used in fair learning rather than the invariant learning. But this distinction is unnecessary as we could define counterfactual fairness more generally, with respect to any transformation of the data, i.e., for a set of transformations~$\{\rho_i\}$,
$$
\mathbb{E}\_{(x, y)} \left[ D_{KL}(f\_{\theta}(\mathbf{x}) || f\_{\theta}(\rho_i (\mathbf{x})) \right] \leq c, \: \text{for all } i.
$$
If the reviewer agrees with us that this is less prone to confusion, we will reformulate the problem in those terms in the revised manuscript.

Reviewer comment (Theory):
About the example of Counterfactual Fairness Constraints — $f _ {\theta}(x)$ and $f_{\theta}(x,z)$ have inputs of different dimensionality. How can you make that happen, for example, in a certain neural network with a certain fixed parameter $\theta$?

Author response:
**(1.4-1.5)** The lack of boldface in the $x$ of the constraints of the learning problem are a typo. Effectively, all $x$ in the problem refer to the feature vector $\mathbf{x}$. As for the input of the function, in retrospect we must agree with the reviewer that our abuse of notation causes confusion. In an attempt to highlight the protected features, we collect them into the separate vector $\mathbf{z}$. In other words, $\mathbf{x} = [\bar{\mathbf{x}}; \mathbf{z}]$, so that we can write $f\_{\theta}(\mathbf{x}) = f\_{\theta}(\bar{\mathbf{x}}, \mathbf{z})$. This is more in keep with the notation used in fair learning rather than the invariant learning. But this distinction is unnecessary as we could define counterfactual fairness more generally, with respect to any transformation of the data, i.e., for a set of transformations~$\{\rho_i\}$,
$$
\mathbb{E}\_{(x, y)} \left[ D_{KL}(f\_{\theta}(\mathbf{x}) || f\_{\theta}(\rho_i (\mathbf{x})) \right] \leq c, \: \text{for all } i.
$$
If the reviewer agrees with us that this is less prone to confusion, we will reformulate the problem in those terms in the revised manuscript.

Reviewer comment (Novelty):
At the end of Section 2, you mentioned that Lagrangian minimizers are not unique and some of them could contain infeasible optimal primal variables. Is this found by you or previous literature? If it is found you, you could provide examples to support your claim and emphasize this claim as one of your novelty. Otherwise, you could cite the papers that prove this claim.

Author response:
**(2)** We would certainly not claim this as contribution of the manuscript as it is a classical result from duality theory [23], [24], [16]. A straightforward example where this happens is in linear programs, where the Lagrangian evaluated at the optimal dual variable $L(\phi, \lambda^\star)$ is a constant (independent of $\phi$). In this case, $\text{argmin}\_{\phi \in \mathcal{F}}\ L(\phi, \lambda^\star) = \mathcal{F}$, which includes many infeasible candidates. This is why, as we mentioned our general response, it is not always possible to extract a \emph{feasible solution} for $(P_p)$ from the optimal dual variables $\lambda^\star_p$ without using primal averaging (for convex problems only) or randomization (for both convex and non-convex problems). Figure 1 (left) in our work further illustrates this phenomenon: though the blue constraint is often infeasible, it turns out to be feasible on average.

Reviewer comment (Writing):
Right after the problem $(P_u)$ at the beginning of Section 3.1, you could mention $\phi\in\mathcal{F}$.

Author response:
**(3)** We will highlight this fact in the revised manuscript.

Reviewer comment (Writing):
It seems that $\phi(\lambda_u)$ (in Assumption 3.2) and $\Phi^*(\lambda_u)$ are the same, right? If yes, you could unify the notations. If not, you could explain the meaning of $\phi(\lambda_p^*)$ in the paper.

Author response:
**(4)** Rather than constantly writing "for $\phi \in \Phi^\star(\lambda)$," we use the shorthand~$\phi(\lambda)$ to denote any element of the set of Lagrange minimizers $\Phi^\star(\lambda)$. In the unparametrized problem, the Lagrangian minimizer is unique, so $\phi^{\star} = \phi^\star(\lambda_u) = \Phi^\star(\lambda_u)$. We agree that this notation can be confusing and that it needs to be further emphasized and explained in the manuscript. We will carefully review the notation in the revised version of the paper.

Reviewer comment (Writing):
In paragraph 2 of Section 3.2, it may be clearer to change ''We will first provide a result assuming this curvature is known'' to something like ''We will first provide a result with the following assumption on the knowledge of this curvature'', so readers can relate to Assumption 3.4.

Author response:
**(5-6)** We appreciate the reviewer suggestions and will incorporate these together with additional clarifications in our revision.

Reviewer comment (Presentation):
After ''The second one captures the effect of parametrizing the hypothesis class for a fixed dual variable.'', you could add something like ''We will elaborate these perturbations in Section 3.3.1 and Section 3.3.2 respectively.''

Author response:
**(5-6)** We appreciate the reviewer suggestions and will incorporate these together with additional clarifications in our revision.

Reviewer comment (Reproducibility):
In the experiment — What negative likelihood function $\ell _ 0$ do you use (or the distribution of $x, y$)? What function $f _ {\theta}$ do you use?

Author response:
**(7.1)** We use the multinomial log-likelihood (as shown when we introduced the counterfactual fairness problem), also known as the cross-entropy loss. We will fix this typo and clarify the loss function in the revised manuscript. As we state at the beginning of Section 5, we take $f\_{\theta}$ to be a "two-layer neural network with 64 nodes and sigmoid activations."

Reviewer comment (Theory):
the bound on $||\ell(f_\theta(\lambda_p^*))- \ell(\phi)||$ contains $||\lambda_p^*||1$, which could be large when the number of coinstraints $m$ is large. When $||\lambda_p^*||$ is large, we cannot assert that $f_\theta(\lambda^*_p)$ is almost feasible. Is there any way to show that $||\lambda_p^*||_1$ is small under some assumptions?

Author response:
Yes. Consider that there exists a parameter $\theta_0$ and a strictly positive constant $C>0$ such that for all constraints $i$ we have that $\ell_i(f_{\theta_0}) \leq -C$. We then have that 
\begin{equation}
\tag{1}
\\| \lambda^{*}_p \\|_1 \leq \frac{\ell_0(f\_{\theta_0})-D^{\star}_p }{C} .
\end{equation}
This is a standard result in duality theory [8] that we have verified holds for the nonconvex optimization problems that we study in this paper. We will incorporate this result and its proof in the final manuscript.  

We point out that the existence of $\theta_0$ and $C>0$ satisfying $\ell_i(f_{\theta_0}) \leq -C$ is not a new assumption. We already require strict feasibility in Assumption 3.2. We simply did not name the strict feasibility constant $C$. Further notice that the bound in (1) does not depend on the number of constraints involved in the optimization problem, but only on the properties of a strictly feasible point $\theta_0$.

Reviewer comment (Theory):
The authors considered $f_\theta(\lambda_p^*)$, the optimal lagrangian minimzer of $P_p$. However, one can only get $f_\theta(\lambda_p^*)$ after infinite iterations. It would be more interesting if the authors can provide some information on the feasibility of $f_\theta(\lambda(T))$. In particular, how close can $f_\theta(\lambda(T))$ and $f_\theta(\lambda_p^*)$ be?

Author response:
This is a good point. Let us point out that the algorithm we propose in this paper is a standard stochastic dual gradient ascent algorithm in which dual iterates are moving along stochastic gradients of the dual function. Since the dual function is concave (dual functions are always concave for constrained minmization problems, convex or not), convergence properties of dual iterates are therefore well understood. They follow from standard convergence results of stochastic gradient ascent for concave functions [12], [20].

In this dual gradient ascent algorithm primal iterates are a byproduct. They are computed as Lagrangian minimizers associated with dual iterates. The main goal of our paper is to study the convergence of these primal iterates, which, as we outline in the general response, is not a given. Even in the case of convex optimization problems, it is not always true that primal iterates of Alg. 1 converge to optimal primal variables. 

Our paper provides a novel analysis for the convergence of primal iterates in (nonconvex) learning problems. To do so we develop novel analyses and proof techniques relying on the comparison of the problem of interest with a convex reference given by the unparameterized constrained learning problem. We think that it is best for readers to keep the paper focused on this contribution and that, consequently, finite time results are a topic of future research.

Reviewer comment (Theory):
The author measured the distance between $\ell(f_\theta(\lambda_p^*))$ and $\ell(\phi^*)$. However, $\phi^*$ doesnot necessarily belong to $F_\theta$. It seems a feasible solution from $F_\theta$ should be a better option and perhaps will bring us better bounds.

Author response:
This is an interesting point. The bounds that we provide in Theorem 3.5 and Corollary 3.9 are of the form:

\begin{equation}
\tag{2}
|  \ell_i(f_{\theta}(\lambda^{\star}_p)) - \ell_i(\phi^{\star}) | \leq \zeta, 
\end{equation}

for some specific constant $\zeta$. This equation is a comparison between the loss $\ell_i(f_{\theta}(\lambda^{\star}_p))$ obtained from the Lagrangian minimizers of the *parameterized* Lagrangian with the loss $\ell_i(\phi^{\star})$ of the solution of the *unparameterized* constrained learning problem. 
    
We take it that the reviewer expected to see a comparison between the loss $\ell_i(f\_{\theta}(\lambda^{\star}_p))$ obtained from the Lagrangian minimizers of the *parameterized* Lagrangian with the loss $\ell_i(f\_{\theta}^*))$ of the solution of the *parameterized* constrained learning problem. They are therefore intrigued by the bounds that appear in Theorem 3.5 and Corollary 3.9.
    
The reason for the unexpected appearance of the difference in the left hand side of Corollary 3.9 is the core technical innovation of our proof techniques. The parametrized learning problem with variables $f_\theta$ is non-convex and difficult to analyze. The unparametrized problem with variable $\phi$ is convex and one that we can analyze using functional convex analysis techniques. Our technical approach relies on the fact that these two problems and their duals can be shown to be $O(\nu)$-close to each other if the parametrization is $\nu$-universal. This is the reason why feasibility bounds end up comparing the loss $\ell_i(f\_{\theta}(\lambda^{\star}_p))$ obtained from the Lagrangian minimizers of the *parameterized* Lagrangian with the loss $\ell_i(\phi^{\star})$  of the solution of the *parameterized* constrained learning problem -- as opposed to the loss $\ell_i(f\_{\theta}^{\star}))$ of the solution of the *parameterized* constrained learning problem.

Further notice that since $\phi^*$ is feasible (i.e: $\ell_i(\phi^*)\leq0$) , an upper bound on the infeasibility of $f_{\theta}(\lambda^*_p)$ can be derived directly from (2). Explicitly, $\ell_i(f\_{\theta}(\lambda^*_p))  \leq  \ell_i(f\_{\theta}(\lambda^*_p)) - \ell_i(\phi^*)  \leq \zeta$.

Reviewer comment (Theory):
The paper's title is near-optimal solutions which is misleading. The main results compare the stationary points of constrained/unconstrained problems as well as using the dual learning algorithm. I'd expect optimality in statistical sense where a lower bound is provided for rates of convergence/estimation error, instead of studying the limiting points error, which is less interesting to me.

Author response:
**(1)** While we appreciate the reviewer's point, we respectfully disagree. While it is true that "optimality" has different meanings in statistics and optimization/algorithms, we do not see why the former should take any priority over the latter. Optimization plays as key a role in modern data science and machine learning as statistics, as evidenced by the overwhelming number of theoretical and empirical results that heavily rely on optimization arguments or improvements on optimization algorithms. We do not therefore believe that what we take "near-optimality" to mean in this manuscript, which is in line with standard definitions both in optimization and computer science (e.g., [14]), is in any way misleading. That being said, we are happy to consider any alternative suggestion the reviewer might have.

Reviewer comment (Theory):
Following the previous point, bounding the error of stationary points is often less interesting in terms of optimality, and lacks of theoretical contributions to the community. Perhaps the authors can provide why such analyses is nontrivial/fundamentally harder than other problems and why it is important to understand this first.

Author response:
**(2)** As explained in the general comment, two challenges must be overcome in this non-convex setting. The first one is the lack of strong duality (as pointed out by reviewer rUm3). In other words, the value $D_p^\star$ of the dual problem $(D_p)$ need not be a good approximation of the value $P_p^\star$ of $(P_p)$. The magnitude of this gap was characterized in [7] Proposition 3.3. Naturally, the quantities that characterize the feasibility of primal iterates are present in the duality gap bound of [7]. For example, the duality gap $ P^{\star}_p - D^{\star}_p $ depends linearly on the richness of the parametrization $\nu$.

The second main challenge is that dual ascent methods such as Algorithm 1 (also referred to as best response) need not converge, and even it did, recovering a feasible solution from optimal dual variables may not be possible. As explained in section 2.2 and shown experimentally on Figure 1, the non-uniqueness of Lagrangian minimizers can lead to large oscillations in the feasibility of primal iterates. In the linear programming example of the general comment (which is a convex problem) the primal iterates obtained as a byproduct of the ascent method can severely violate the constraints. Indeed, in this case, the Lagrangian $L(\phi, \lambda^\star)$ is a constant (independent of $\phi$), i.e., $\Phi^\star(\lambda^\star) = \mathcal{F}$. The difficulty of recovering (near-)feasible primal solutions explains the need of randomized solutions (see
e.g, ([10], Theorem 2; [11], Theorem 4.1; [7], Theorem 3). However, randomized predictors are impractical and do not solve problem $(P_p)$. Thus, it is highly relevant to determine conditions under which a near-feasible primal iterate, **which is also near-optimal** can be recovered. Also note that the analysis carried out in this paper involved novel proof techniques (as per reviewer z2M9), such as the use of the curvature of the perturbation function (see section 3.3.2).

Reviewer comment (Experiments):
The numerical experiments seem weak. It is not suprising that there will be a gap and as the number of features grow the model approximation error is smaller. I'm not seeing how this directly corrobates the theory. Ideally I would like to see (i) a hypothesis emerged from the theory (ii) an experiment that validates the hypothesis.

Author response:
**(3)** We respectfully disagree. The experiments in this paper clearly illustrate the fact that richer parametrizations lead to better properties of Alg. 1 in terms of the constraint violation of its iterates. This behavior, suggested by our theorems, is not straightforward. Indeed, this paper analyses the optimization error when solving constrained learning problems and not the estimation or approximation error~(see general response). Hence, while it is indeed expected that richer parametrizations will lead to lower approximation error, it is certainly not straightforward that richer models make optimization easier. While better parametrizations do provide more feasible models to choose from, they also contain more infeasible ones. Finding solutions requires handling a much larger class of hypothesis, so that we would actually expected the optimization to be harder. Our results, however, prove that richer parametrizations make it easier for the algorithm to find a feasible solution. In fact, not only a feasible solutions, but one that is also near optimal, which is not straightforward (as per our previous answer). This conclusion from Proposition 4.2 is illustrated in Figure 2(right). Note that this plot does not represent the approximation (test) error, but the constraint violation. We will work on improving the description and axis labels of these plots to avoid confusion in the future.

Reviewer comment (Evaluation):
The interpretation of the results is not clearly stated. In an attempt to be perfectly general, it is hard to state exactly how the results apply to any particular learning problem.

Author response:
We thank the reviewer for the detailed feedback and for highlighting the novelty and rigour of our work. Next, we address the weaknesses they point out (i.e, interpretation of the results and strictness of assumptions) and answer the concerns they raised.

**Weaknesses**

**(1)** As we mention in our general response, the issue of neural networks vs poorly chosen kernel relates to the approximation (ii) [and perhaps generalization (iii)] error. In this manuscript, we study the algorithmic error (iii) that only considers how good a solution of $(P_p)$ can be obtained using Alg. 1. Whether $(P_p)$ yields good solutions for learning problem, i.e., errors (ii)-(iii), is tackled in, e.g., [7], [15]. In the case of a single poorly chosen kernel, there is no algorithmic error (iii): either the one kernel is feasible or the problem has no solution. This is reflected in our results ($\lambda \to \infty$ if $(P_p)$ is infeasible).

Things are more interesting in the neural network case. Indeed, while we do not analyze the approximation error (ii), the richness of the parametrization still plays a role in our results. Indeed, we show that more expressive models (Assumption 3) have a positive impact on the algorithm behavior. In other words, there are stronger guarantees for Alg. 1 to find near-feasible solutions of $(P_p)$ when using larger NNs. This point is illustrated in Figure 2(right), which shows the constraint violation as a function of the model capacity. This is not straightforward: while better parametrizations do provide more feasible models to choose from, they also contain more infeasible ones. Finding feasible solutions requires handling a much larger class of hypothesis, so we could actually expected the optimization to be harder. Combined with the duality gap bounds from [7], [15], this implies that Alg. 1 finds solutions that are not only near-feasible, but also near-optimal. This is not straightforward considering the lack of strong duality due to non-convexity and the challenges in proving convergence of primal iterates without randomization (see general response).

We will add new discussions of these interpretations and highlight these conclusions in the revised version of the manuscript.

Reviewer comment (Experiments):
One would think that algorithms based on neural networks would perform quite differently than a kernel-based approach using a single poorly chosen kernel, yet the results don't seem to account for such differences in any way.

Author response:
As we mention in our general response, the issue of neural networks vs poorly chosen kernel relates to the approximation (ii) [and perhaps generalization (iii)] error. In this manuscript, we study the algorithmic error (iii) that only considers how good a solution of $(P_p)$ can be obtained using Alg. 1. Whether $(P_p)$ yields good solutions for learning problem, i.e., errors (ii)-(iii), is tackled in, e.g., [7], [15]. In the case of a single poorly chosen kernel, there is no algorithmic error (iii): either the one kernel is feasible or the problem has no solution. This is reflected in our results ($\lambda \to \infty$ if $(P_p)$ is infeasible).

Things are more interesting in the neural network case. Indeed, while we do not analyze the approximation error (ii), the richness of the parametrization still plays a role in our results. Indeed, we show that more expressive models (Assumption 3) have a positive impact on the algorithm behavior. In other words, there are stronger guarantees for Alg. 1 to find near-feasible solutions of $(P_p)$ when using larger NNs. This point is illustrated in Figure 2(right), which shows the constraint violation as a function of the model capacity. This is not straightforward: while better parametrizations do provide more feasible models to choose from, they also contain more infeasible ones. Finding feasible solutions requires handling a much larger class of hypothesis, so we could actually expected the optimization to be harder. Combined with the duality gap bounds from [7], [15], this implies that Alg. 1 finds solutions that are not only near-feasible, but also near-optimal. This is not straightforward considering the lack of strong duality due to non-convexity and the challenges in proving convergence of primal iterates without randomization (see general response).

We will add new discussions of these interpretations and highlight these conclusions in the revised version of the manuscript.

Reviewer comment (Theory):
The lack of clarity in applicability makes it hard to understand if the 7 (mostly regularity) assumptions are valid and what the corresponding parameters are.

Author response:
Our results effectively depend on five assumptions (since Assumption 3.4 is implied by 3.1, 3.6, and 3.7 and number 3.5 is a theorem). We summarize them below:

- **Assumption 3.1**: $\ell_i$ are convex and Lipschitz continuous. $\ell_0$ is strongly convex
- **Assumption 3.2**: there exists a strictly feasible $\phi$, i.e., $\ell_i(\phi) \leq C$ for all~$i$ and some $C > 0$
- **Assumption 3.3**: For all $\phi \in \mathcal{F}$, there exists $\theta \in \Theta$ such that $\| \phi - f_{\theta} \|_{L_2} \leq \nu$
- **Assumption 3.6**: $\ell_i$ are smooth (Lipschitz gradients)
- **Assumption 3.7**: The Jacobian $D_{\phi} \mathbf{\ell}( \phi^{\star})$ has full-row rank, where $D_{\phi}$ denotes the Fréchet derivative

As the reviewer notes, these assumptions are mostly regularity assumptions over which a practitioner has full control. Indeed, Assumptions 3.1 and 3.6 impose mild restrictions on the losses, namely, that they be Lipschitz continuous, convex, and smooth. These are widely used to prove convergence results for gradient descent algorithms even in the convex setting and are satisfied by typical losses used in ML when, e.g., the hypothesis class $\mathcal{F}$ is compact. Assumption 3.3 is always satisfied for some finite $\nu$ and describes the richness of the parametrization. A detailed description of this assumption was provided in section 3.1 as well as our previous answer.

Assumptions 3.2 and 3.7 are closely related to constraint qualifications widely used in the analysis of Alg. 1 in convex settings. Indeed, Assumption 3.2 is a stricter version of the well-known Slater's condition from convex optimization. It guarantees that $(P_u)$ and $(P_p)$ are well-posed. For overparametrized models, this assumption is not strict. Assumption 3.7, on the other hand, is not straightforward to satisfy at first sight. It is, however, a typical assumption used to derive duality results in convex optimization known as "linear independence constraint qualification" or LICQ [16]. In fact, it is the weakest constraint qualifications for convex optimization. This suggests that it may be possible to replace Assumption 3.7 by a stricter version of Assumption 3.2, although this is not immediate. We therefore choose to state the weakest assumptions under which our results would hold.
    
We will expand the description of these assumptions already present in Section 3.1 to highlight their mildness and limitations in the revised version.