Original: Adaptive Inexact Sequential Quadratic Programming via Iterative Randomized Sketching
Revised: Adaptive Inexact Sequential Quadratic Programming via Iterative Randomized Sketching

Original: Abstract
Revised: Abstract

Original: We consider solving nonlinear optimization problems with equality constraints.
Revised: We consider solving nonlinear optimization problems with equality constraints.

Original: We propose a randomized algorithm based on sequential quadratic programming (SQP) with a differentiable exact augmented Lagrangian as the merit function.
Revised: We propose a randomized algorithm based on sequential quadratic programming (SQP) with a differentiable exact augmented Lagrangian as the merit function.

Original: In each SQP iteration, we solve the Newton system inexactly via iterative randomized sketching.
Revised: In each SQP iteration, we solve the Newton system inexactly via iterative randomized sketching.

Original: The accuracy of the inexact solution and the penalty parameter of the augmented Lagrangian are adaptively controlled in the algorithm to ensure that the inexact random search direction is a descent direction of the augmented Lagrangian.
Revised: The accuracy of the inexact solution and the penalty parameter of the augmented Lagrangian are adaptively controlled in the algorithm to ensure that the inexact random search direction is a descent direction of the augmented Lagrangian.

Original: This allows us to establish global convergence almost surely.
Revised: This allows us to establish global convergence almost surely.

Original: Moreover, we show that a unit stepsize is admissible for the inexact search direction provided the iterate lies in a neighborhood of the solution.
Revised: Moreover, we show that a unit stepsize is admissible for the inexact search direction provided the iterate lies in a neighborhood of the solution.

Original: Based on this result, we show that the proposed algorithm exploits local linear convergence.
Revised: Based on this result, we show that the proposed algorithm exploits local linear convergence.

Original: We apply the algorithm on benchmark nonlinear problems in CUTEst test set and on constrained logistic regression with datasets from LIBSVM to demonstrate its superior performance.
Revised: We apply the algorithm on benchmark nonlinear problems in CUTEst test set and on constrained logistic regression with datasets from LIBSVM to demonstrate its superior performance.

Original: 1. Introduction
Revised: 1. Introduction

Original: We consider the nonlinear equality-constrained optimization problem
Revised: We consider the nonlinear equality-constrained optimization problem

Original: [Equation]
Revised: [Equation]

Original: where f : R n → R is the objective function and c : R n → R m are equality constraints.
Revised: where f : R n → R is the objective function and c : R n → R m are equality constraints.

Original: There exist numerous methods for solving Problem (1.1), including projected first- and second-order methods, penalty methods, augmented Lagrangian methods and sequential quadratic programming (SQP).
Revised: There exist numerous methods for solving Problem (1.1), including projected first- and second-order methods, penalty methods, augmented Lagrangian methods and sequential quadratic programming (SQP).

Original: In this paper, we focus on solving (1.1) via SQP, which is one of the leading second-order methods for constrained optimization problems [9, 10, 14].
Revised: In this paper, we focus on solving (1.1) via SQP, which is one of the leading second-order methods for constrained optimization problems [9, 10, 14].

Original: The algorithms in this class typically enjoy global convergence guarantees and require a few iterations to find a local solution.
Revised: The algorithms in this class typically enjoy global convergence guarantees and require a few iterations to find a local solution.

Original: However, the computational cost of SQP algorithms is dominated by solving one (or more) Newton system in each iteration, which can be prohibitive for large-scale problems.
Revised: However, the computational cost of SQP algorithms is dominated by solving one (or more) Newton system in each iteration, which can be prohibitive for large-scale problems.

Original: To reduce the per-iteration computational cost, [4] proposed an inexact SQP algorithm where, in each iteration, the Newton system is approximately solved using a deterministic iterative solver and the stepsizes are chosen based on a penalized merit function.
Revised: To reduce the per-iteration computational cost, [4] proposed an inexact SQP algorithm where, in each iteration, the Newton system is approximately solved using a deterministic iterative solver and the stepsizes are chosen based on a penalized merit function.

Original: With suitable conditions on the quality of the inexact solution, the authors showed that the inexact search direction is still a descent direction of the merit function and the algorithm enjoys global convergence.
Revised: With suitable conditions on the quality of the inexact solution, the authors showed that the inexact search direction is still a descent direction of the merit function and the algorithm enjoys global convergence.

Original: Despite the solid theoretical underpinnings, the algorithm of [4] suffers from few drawbacks.
Revised: Despite the solid theoretical underpinnings, the algorithm of [4] suffers from few drawbacks.

Original: First, for each
Revised: First, for each

Original: SQP iteration, the algorithm relies on a few fixed tuning parameters ( κ 1 , κ 2 , ϵ , β ) for bounding the residuals of the iterative solver.
Revised: SQP iteration, the algorithm relies on a few fixed tuning parameters ( κ 1 , κ 2 , ϵ , β ) for bounding the residuals of the iterative solver.

Original: These parameters may substantially affect the performance of the algorithm and have to be chosen carefully.
Revised: These parameters may substantially affect the performance of the algorithm and have to be chosen carefully.

Original: In particular, a tighter residual bound will lead to more inner loop iterations to compute a more precise step.
Revised: In particular, a tighter residual bound will lead to more inner loop iterations to compute a more precise step.

Original: However, the cost of more inner loop iterations must be balanced against a possible decrease in the outer loop iterations for finding the local solution.
Revised: However, the cost of more inner loop iterations must be balanced against a possible decrease in the outer loop iterations for finding the local solution.

Original: Second, the algorithm uses a nonsmooth merit function ϕ
Revised: Second, the algorithm uses a nonsmooth merit function ϕ

Original: π (
Revised: π (

Original: [Equation]
Revised: [Equation]

Original: when performing the line search, which is known to cause the Maratos effect—a unit stepsize may not be accepted near the solution.
Revised: when performing the line search, which is known to cause the Maratos effect—a unit stepsize may not be accepted near the solution.

Original: Such an effect leads to a slow local convergence [5].
Revised: Such an effect leads to a slow local convergence [5].

Original: Third, the local behavior of that algorithm has not been rigorously analyzed.
Revised: Third, the local behavior of that algorithm has not been rigorously analyzed.

Original: In this paper, we propose a randomized SQP algorithm to solve Problem (1.1) in which the
Revised: In this paper, we propose a randomized SQP algorithm to solve Problem (1.1) in which the

Original: Newton system in the inner loop is solved using the iterative randomized sketching (IRS) [8].
Revised: Newton system in the inner loop is solved using the iterative randomized sketching (IRS) [8].

Original: Thus, the proposed method could be seen as a randomized extension of [4].
Revised: Thus, the proposed method could be seen as a randomized extension of [4].

Original: Furthermore, instead of using fixed bound to control the accuracy of the inexact search direction throughout all SQP iterations, the proposed method adaptively controls the accuracy of the inexact solution to balance between the number of inner and outer loop iterations whilst the method achieves fast local convergence.
Revised: Furthermore, instead of using fixed bound to control the accuracy of the inexact search direction throughout all SQP iterations, the proposed method adaptively controls the accuracy of the inexact solution to balance between the number of inner and outer loop iterations whilst the method achieves fast local convergence.

Original: We use a differentiable exact augmented Lagrangian as the merit function of the form
Revised: We use a differentiable exact augmented Lagrangian as the merit function of the form

Original: [Equation]
Revised: [Equation]

Original: where L ( x , λ ) = f ( x ) + λ T c ( x ) is the Lagrangian function of Problem (1.1) with λ ∈ R m being the Lagrangian multipliers, and η = ( η 1 , η 2 ) is the penalty parameter.
Revised: where L ( x , λ ) = f ( x ) + λ T c ( x ) is the Lagrangian function of Problem (1.1) with λ ∈ R m being the Lagrangian multipliers, and η = ( η 1 , η 2 ) is the penalty parameter.

Original: The benefit of using an exact penalty function is that a stationary point ( x ⋆ , λ ⋆ ) of (1.2) is also a stationary point of
Revised: The benefit of using an exact penalty function is that a stationary point ( x ⋆ , λ ⋆ ) of (1.2) is also a stationary point of

Original: Problem (1.1) and vice versa, provided that η 1 is sufficiently large and η 2 is sufficiently small [2,
Revised: Problem (1.1) and vice versa, provided that η 1 is sufficiently large and η 2 is sufficiently small [2,

Original: Proposition 4.15].
Revised: Proposition 4.15].

Original: Further, the smoothness of the merit function in (1.2) effectively overcomes the
Revised: Further, the smoothness of the merit function in (1.2) effectively overcomes the

Original: Maratos effect [3].
Revised: Maratos effect [3].

Original: We emphasize three novelties of the proposed algorithm.
Revised: We emphasize three novelties of the proposed algorithm.

Original: First, we use the iterative randomized sketching [8] to compute an inexact solution of the Newton system.
Revised: First, we use the iterative randomized sketching [8] to compute an inexact solution of the Newton system.

Original: Projecting a large Newton system into a smaller one and obtaining an approximate solution leads to large computational savings [8, 16, 17].
Revised: Projecting a large Newton system into a smaller one and obtaining an approximate solution leads to large computational savings [8, 16, 17].

Original: Second, the algorithm adaptively selects a parameter that controls a bound on the residuals when accepting the search direction.
Revised: Second, the algorithm adaptively selects a parameter that controls a bound on the residuals when accepting the search direction.

Original: As a result, the inexact solution of the Newton system is a descent direction of the merit function, and is accurate enough to guarantee the global and local linear convergence of the algorithm.
Revised: As a result, the inexact solution of the Newton system is a descent direction of the merit function, and is accurate enough to guarantee the global and local linear convergence of the algorithm.

Original: Empirically, our adaptive algorithm results in smaller KKT residuals (the sum of the feasibility error and the optimality error) and fewer gradient evaluations.
Revised: Empirically, our adaptive algorithm results in smaller KKT residuals (the sum of the feasibility error and the optimality error) and fewer gradient evaluations.

Original: Third, despite the randomness in the inexact search direction brought by the randomized solver, we establish the almost sure global convergence.
Revised: Third, despite the randomness in the inexact search direction brought by the randomized solver, we establish the almost sure global convergence.

Original: Furthermore, we show that the algorithm locally selects a unit stepsize even with the adaptive step acceptance condition, which leads to a local linear convergence rate.
Revised: Furthermore, we show that the algorithm locally selects a unit stepsize even with the adaptive step acceptance condition, which leads to a local linear convergence rate.

Original: Such a local result complements the existing literature on inexact SQP algorithms.
Revised: Such a local result complements the existing literature on inexact SQP algorithms.

Original: 2. Method
Revised: 2. Method

Original: We propose an adaptive inexact SQP algorithm that uses iterative randomized sketching to solve the
Revised: We propose an adaptive inexact SQP algorithm that uses iterative randomized sketching to solve the

Original: Newton system in the inner loop.
Revised: Newton system in the inner loop.

Original: We use ∥ · ∥ to denote the ℓ 2 norm for vectors and the operator norm for matrices.
Revised: We use ∥ · ∥ to denote the ℓ 2 norm for vectors and the operator norm for matrices.

Original: At the k -th outer iteration, we let f k
Revised: At the k -th outer iteration, we let f k

Original: =
Revised: =

Original: f
Revised: f

Original: ( x k ) , etc., to simplify the notation.
Revised: ( x k ) , etc., to simplify the notation.

Original: When a constraint qualification holds, the first-order necessary conditions for x ⋆ to be a solution to Problem (1.1) are that there exist multipliers λ
Revised: When a constraint qualification holds, the first-order necessary conditions for x ⋆ to be a solution to Problem (1.1) are that there exist multipliers λ

Original: ⋆ such that
Revised: ⋆ such that

Original: [Equation]
Revised: [Equation]

Original: where G ( x ) = ∇ T c ( x ) = ( ∇ c 1 ( x ) , . .
Revised: where G ( x ) = ∇ T c ( x ) = ( ∇ c 1 ( x ) , . .

Original: , ∇ c m ( x ))
Revised: , ∇ c m ( x ))

Original: T ∈ R m × n is the constraint Jacobian.
Revised: T ∈ R m × n is the constraint Jacobian.

Original: In each outer iteration k , the SQP algorithm finds the search direction (∆ x k , ∆ λ k ) by solving the following
Revised: In each outer iteration k , the SQP algorithm finds the search direction (∆ x k , ∆ λ k ) by solving the following

Original: Newton system
Revised: Newton system

Original: [Equation]
Revised: [Equation]

Original: where B ( x , λ ) is the Hessian of the Lagrangian ∇ 2 x L ( x , λ )
Revised: where B ( x , λ ) is the Hessian of the Lagrangian ∇ 2 x L ( x , λ )

Original: =
Revised: =

Original: H ( x , λ ) or its symmetric perturbation.
Revised: H ( x , λ ) or its symmetric perturbation.

Original: Let Γ k = (cid:18)
Revised: Let Γ k = (cid:18)

Original: B k
Revised: B k

Original: G Tk
Revised: G Tk

Original: G
Revised: G

Original: [Equation]
Revised: [Equation]

Original: ∈ R ( n + m ) × ( n + m ) and we rewrite the Newton system (2.2) by
Revised: ∈ R ( n + m ) × ( n + m ) and we rewrite the Newton system (2.2) by

Original: [Equation]
Revised: [Equation]

Original: Instead of finding the exact Newton direction (∆ x k , ∆ λ k ) , we apply the iterative randomized sketching to obtain an inexact solution ( ˜∆ x k , ˜∆ λ k ) to (2.3).
Revised: Instead of finding the exact Newton direction (∆ x k , ∆ λ k ) , we apply the iterative randomized sketching to obtain an inexact solution ( ˜∆ x k , ˜∆ λ k ) to (2.3).

Original: In particular, we let S ∈ R ( n + m ) ×
Revised: In particular, we let S ∈ R ( n + m ) ×

Original: d be a random sketch matrix which has some probability distribution P and for each outer iteration k and inner iteration j , we specify each random matrix by S k,j ∼ S .
Revised: d be a random sketch matrix which has some probability distribution P and for each outer iteration k and inner iteration j , we specify each random matrix by S k,j ∼ S .

Original: For j -th inexact solution ( ˜∆ x k,j , ˜∆ λ k,j ) , we define the residual vectors of the Newton system by
Revised: For j -th inexact solution ( ˜∆ x k,j , ˜∆ λ k,j ) , we define the residual vectors of the Newton system by

Original: [Equation]
Revised: [Equation]

Original: Then the inner loop iteration updates the solution as
Revised: Then the inner loop iteration updates the solution as

Original: [Equation]
Revised: [Equation]

Original: where W k,j = Γ Tk S k,j (cid:16) S Tk,j Γ k Γ Tk S k,j (cid:17) − 1 S Tk,j ∈ R ( n + m ) × ( n + m ) .
Revised: where W k,j = Γ Tk S k,j (cid:16) S Tk,j Γ k Γ Tk S k,j (cid:17) − 1 S Tk,j ∈ R ( n + m ) × ( n + m ) .

Original: Now we define
Revised: Now we define

Original: [Equation]
Revised: [Equation]

Original: where Υ k =
Revised: where Υ k =

Original: ∥
Revised: ∥

Original: B k ∥ ∨ ∥ G k ∥ ∨ ∥
Revised: B k ∥ ∨ ∥ G k ∥ ∨ ∥

Original: H k ∥ and Ψ k is defined in Lemma 6.
Revised: H k ∥ and Ψ k is defined in Lemma 6.

Original: At each outer iteration k , we force the adaptive parameter δ k , which controls the accuracy of the inexact solution of (2.3), to be smaller than δ trial k .
Revised: At each outer iteration k , we force the adaptive parameter δ k , which controls the accuracy of the inexact solution of (2.3), to be smaller than δ trial k .

Original: This procedure ensures the algorithm selects a unit stepsize locally, so that it enjoys the local linear convergence near a stationary point of Problem (1.1).
Revised: This procedure ensures the algorithm selects a unit stepsize locally, so that it enjoys the local linear convergence near a stationary point of Problem (1.1).

Original: For the simplicity of notation, we drop the inner iteration j from ( ˜∆ x k,j , ˜∆ λ k,j ) and r k,j when we generally refer to the inexact search direction and residual vector.
Revised: For the simplicity of notation, we drop the inner iteration j from ( ˜∆ x k,j , ˜∆ λ k,j ) and r k,j when we generally refer to the inexact search direction and residual vector.

Original: The following condition describes when a search direction will be accepted.
Revised: The following condition describes when a search direction will be accepted.

Original: Step Acceptance Condition.
Revised: Step Acceptance Condition.

Original: Given η 1 ,k , η 2 ,k > 0 and 0 < δ k ≤ δ trial k , a step ( ˜∆ x k , ˜∆ λ k ) that is computed via (2.5) is acceptable if
Revised: Given η 1 ,k , η 2 ,k > 0 and 0 < δ k ≤ δ trial k , a step ( ˜∆ x k , ˜∆ λ k ) that is computed via (2.5) is acceptable if

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: In each outer iteration k , we first update the inexact search direction by (2.5) until the residual at the inner iteration j satisfies (2.7) with a given δ k .
Revised: In each outer iteration k , we first update the inexact search direction by (2.5) until the residual at the inner iteration j satisfies (2.7) with a given δ k .

Original: Then, we check if the inexact search direction is a descent direction of (1.2); that is, whether (2.8) is satisfied for given ( η 1 ,k , η 2 ,k ) .
Revised: Then, we check if the inexact search direction is a descent direction of (1.2); that is, whether (2.8) is satisfied for given ( η 1 ,k , η 2 ,k ) .

Original: If it does not satisfy (2.8), we increase η 1 ,k , and decrease η 2 ,k and δ k as
Revised: If it does not satisfy (2.8), we increase η 1 ,k , and decrease η 2 ,k and δ k as

Original: [Equation]
Revised: [Equation]

Original: where ν > 1 is a given constant.
Revised: where ν > 1 is a given constant.

Original: We repeat the above two steps until we find an inexact search direction which satisfies (2.7) and (2.8) with appropriate ( η 1 ,k , η 2 ,k , δ k ) .
Revised: We repeat the above two steps until we find an inexact search direction which satisfies (2.7) and (2.8) with appropriate ( η 1 ,k , η 2 ,k , δ k ) .

Original: We design this scheme using double while loops in Algorithm 1.
Revised: We design this scheme using double while loops in Algorithm 1.

Original: The stepsize α k is selected to satisfy the Armijo condition
Revised: The stepsize α k is selected to satisfy the Armijo condition

Original: [Equation]
Revised: [Equation]

Original: and the iterate is updated as
Revised: and the iterate is updated as

Original: [Equation]
Revised: [Equation]

Original: Our Algorithm 1 is presented in Appendix C.
Revised: Our Algorithm 1 is presented in Appendix C.

Original: 3. Convergence Analysis
Revised: 3. Convergence Analysis

Original: We now study well-posedness of Algorithm 1, and establish global and local linear convergence guarantees.
Revised: We now study well-posedness of Algorithm 1, and establish global and local linear convergence guarantees.

Original: We emphasize that the randomness plays a key role in the analysis since the inexact search direction is calculated by the iterative randomized solver.
Revised: We emphasize that the randomness plays a key role in the analysis since the inexact search direction is calculated by the iterative randomized solver.

Original: Compared to an algorithm that uses a deterministic iterative solver, our inexact search direction is stochastic.
Revised: Compared to an algorithm that uses a deterministic iterative solver, our inexact search direction is stochastic.

Original: As a result, all the components of the algorithm that are affected by the search direction are also random; for example, (2.7), (2.8), and (2.10).
Revised: As a result, all the components of the algorithm that are affected by the search direction are also random; for example, (2.7), (2.8), and (2.10).

Original: Our analysis relies on the following assumption.
Revised: Our analysis relies on the following assumption.

Original: Assumption 1 All the iterates { x k } k ≥ 0 belong to an open convex set X .
Revised: Assumption 1 All the iterates { x k } k ≥ 0 belong to an open convex set X .

Original: The objective function f is twice continuously differentiable and bounded over X .
Revised: The objective function f is twice continuously differentiable and bounded over X .

Original: Its gradient ∇ f and Hessian ∇ 2 f are Lipschitz continuous and bounded over X .
Revised: Its gradient ∇ f and Hessian ∇ 2 f are Lipschitz continuous and bounded over X .

Original: The constraint function c is twice continuously differentiable, Lipschitz continuous, and bounded over X .
Revised: The constraint function c is twice continuously differentiable, Lipschitz continuous, and bounded over X .

Original: Its Jacobian G and Hessian of each coordinate are Lipschitz continuous and bounded below over X .
Revised: Its Jacobian G and Hessian of each coordinate are Lipschitz continuous and bounded below over X .

Original: Assumption 2 The Jacobian matrices { G k } k ≥ 0 have full row rank.
Revised: Assumption 2 The Jacobian matrices { G k } k ≥ 0 have full row rank.

Original: There exist constants ξ B , Υ B > 0 , such that, for any outer iteration k ≥ 0 , z T B k z ≥ ξ B ∥ z ∥ 2 for any z ∈ { z : G k z = 0 } and ∥ B k ∥ ≤ Υ B . 
Revised: There exist constants ξ B , Υ B > 0 , such that, for any outer iteration k ≥ 0 , z T B k z ≥ ξ B ∥ z ∥ 2 for any z ∈ { z : G k z = 0 } and ∥ B k ∥ ≤ Υ B . DAPTIVE NEXACT

Original: Assumption 

Revised: Assumption

Original: The random sketch matrix S satisfies P ( S T z ̸ = 0 ) > 0 for any z ∈ R n + m \{ 0 } .
Revised: The random sketch matrix S satisfies P ( S T z ̸ = 0 ) > 0 for any z ∈ R n + m \{ 0 } .

Original: For any outer and inner iteration k, j ≥ 0 , S k,j i.i.d.
Revised: For any outer and inner iteration k, j ≥ 0 , S k,j i.i.d.

Original: ∼ S .
Revised: ∼ S .

Original: Assumption 1 does not make any assumptions about the set Λ that contains the dual iterates { λ k } k ≥ 0 .
Revised: Assumption 1 does not make any assumptions about the set Λ that contains the dual iterates { λ k } k ≥ 0 .

Original: The boundedness of Λ can be proven based on the algorithm itself; see Lemma 13 in
Revised: The boundedness of Λ can be proven based on the algorithm itself; see Lemma 13 in

Original: Appendix A. Assumption 2 implies that Γ k in (2.3) is invertible.
Revised: Appendix A. Assumption 2 implies that Γ k in (2.3) is invertible.

Original: Therefore, for any outer iteration k , the Newton system (2.3) has a unique solution.
Revised: Therefore, for any outer iteration k , the Newton system (2.3) has a unique solution.

Original: This is a standard assumption in the SQP literature [3].
Revised: This is a standard assumption in the SQP literature [3].

Original: Assumption 3 is used specifically to establish the well-posedness of Algorithm 1.
Revised: Assumption 3 is used specifically to establish the well-posedness of Algorithm 1.

Original: In
Revised: In

Original: Lemma 7, we first define the subsequence of the inner iteration { j l } l ≥ 0 where the reduction of the error step occurs, and show that the event A k = ∩
Revised: Lemma 7, we first define the subsequence of the inner iteration { j l } l ≥ 0 where the reduction of the error step occurs, and show that the event A k = ∩

Original: Ll =1 { j l < ∞} happens with probability 1.
Revised: Ll =1 { j l < ∞} happens with probability 1.

Original: Thus, conditioned on the event A k , the error linearly decays in those iterations (see Lemma 8).
Revised: Thus, conditioned on the event A k , the error linearly decays in those iterations (see Lemma 8).

Original: Lemmashows that for each outer iteration k , conditioned on the event
Revised: Lemmashows that for each outer iteration k , conditioned on the event

Original: A k , almost surely, there exists finite inner iteration such that the first component in Step Acceptance Condition (2.7) is satisfied.
Revised: A k , almost surely, there exists finite inner iteration such that the first component in Step Acceptance Condition (2.7) is satisfied.

Original: We denote this event as B k .
Revised: We denote this event as B k .

Original: In Lemma 10, we show that conditioned on the event A k ∩ B k , the second component in Step Acceptance Condition (2.8) is satisfied.
Revised: In Lemma 10, we show that conditioned on the event A k ∩ B k , the second component in Step Acceptance Condition (2.8) is satisfied.

Original: Thus, Lemma 9 and 10 imply that the double while loop in Algorithm 1 terminates in finite time.
Revised: Thus, Lemma 9 and 10 imply that the double while loop in Algorithm 1 terminates in finite time.

Original: Furthermore, Lemma 11 shows that all adaptive parameters ( η 1 , η 2 , δ ) will be fixed at some values after a number of outer iterations.
Revised: Furthermore, Lemma 11 shows that all adaptive parameters ( η 1 , η 2 , δ ) will be fixed at some values after a number of outer iterations.

Original: The formal statements of Lemma 7–11 are presented in Appendix A. Finally, we establish the global convergence of Algorithm 1 in Theorem 1.
Revised: The formal statements of Lemma 7–11 are presented in Appendix A. Finally, we establish the global convergence of Algorithm 1 in Theorem 1.

Original: Theorem 1 (Global convergence) Suppose Assumption 1, 2, 3 hold for the iterates { ( x k , λ k ) } k ≥generated by Algorithm 1.
Revised: Theorem 1 (Global convergence) Suppose Assumption 1, 2, 3 hold for the iterates { ( x k , λ k ) } k ≥generated by Algorithm 1.

Original: Then ∥∇L k ∥ → 0 as k → ∞ almost surely.
Revised: Then ∥∇L k ∥ → 0 as k → ∞ almost surely.

Original: Next, we establish local linear convergence guarantees of Algorithm 1.
Revised: Next, we establish local linear convergence guarantees of Algorithm 1.

Original: We first present two additional assumptions that are necessary to the local behaviour analysis.
Revised: We first present two additional assumptions that are necessary to the local behaviour analysis.

Original: Assumption 4 The third derivative of the objective function ∇ 3 f exists and continuous over X .
Revised: Assumption 4 The third derivative of the objective function ∇ 3 f exists and continuous over X .

Original: The third derivative of the constraints ∇ 3 c i exists and continuous over X for all i ∈ { 1 , . . .
Revised: The third derivative of the constraints ∇ 3 c i exists and continuous over X for all i ∈ { 1 , . . .

Original: , m } .
Revised: , m } .

Original: Assumption 5 For any outer iteration k ≥ 0 , ∥
Revised: Assumption 5 For any outer iteration k ≥ 0 , ∥

Original: H k − B k ∥
Revised: H k − B k ∥

Original: =
Revised: =

Original: o (1) .
Revised: o (1) .

Original: Assumption 4 strengthens the condition of the objective function f and constraints c in Assumption 1 to thrice continuously differentiability.
Revised: Assumption 4 strengthens the condition of the objective function f and constraints c in Assumption 1 to thrice continuously differentiability.

Original: For Assumption 4, when using the augmented
Revised: For Assumption 4, when using the augmented

Original: Lagrangian as the merit function (see (1.2)), it is common to assume the existence of third derivatives of f and c i , since the Hessian of the augmented Lagrangian ∇ 2 x L η requires ∇ 3 f and ∇ 3 c i to exist.
Revised: Lagrangian as the merit function (see (1.2)), it is common to assume the existence of third derivatives of f and c i , since the Hessian of the augmented Lagrangian ∇ 2 x L η requires ∇ 3 f and ∇ 3 c i to exist.

Original: The existence of third derivatives is only necessary for analysis, and they are never computed in practice.
Revised: The existence of third derivatives is only necessary for analysis, and they are never computed in practice.

Original: Assumption 5 is standard in the SQP literature and is needed to show local superlinear or quadratic convergence [3].
Revised: Assumption 5 is standard in the SQP literature and is needed to show local superlinear or quadratic convergence [3].

Original: Now we establish local linear convergence of Algorithm 1 in Theorem 2.
Revised: Now we establish local linear convergence of Algorithm 1 in Theorem 2.

Original: Theorem 2 (Local linear convergence) Let ( x ⋆ , λ ⋆ ) be a stationary point of (1.1) .
Revised: Theorem 2 (Local linear convergence) Let ( x ⋆ , λ ⋆ ) be a stationary point of (1.1) .

Original: Suppose Assumption 1, 2, 3, 4, 5 hold for the iterates { ( x k , λ k ) } k ≥ 0 generated by Algorithm 1, and ( x k , λ k ) → ( x ⋆ , λ ⋆ ) .
Revised: Suppose Assumption 1, 2, 3, 4, 5 hold for the iterates { ( x k , λ k ) } k ≥ 0 generated by Algorithm 1, and ( x k , λ k ) → ( x ⋆ , λ ⋆ ) .

Original: Then for all sufficiently large outer iteration k , almost surely, α k = 1 , and
Revised: Then for all sufficiently large outer iteration k , almost surely, α k = 1 , and

Original: [Equation]
Revised: [Equation]

Original: where δ ⋆ be the stabilized value of δ ∈ (0 , 1) .
Revised: where δ ⋆ be the stabilized value of δ ∈ (0 , 1) .

Original: Proof of Theorem 1 and 2 are given in Appendix B.
Revised: Proof of Theorem 1 and 2 are given in Appendix B.

Original: 4. Experiments
Revised: 4. Experiments

Original: We implement three inexact SQP algorithms to solve benchmark nonlinear problems in CUTEst test set [7] and solve constrained logistic regression with datasets from LIBSVM [6].
Revised: We implement three inexact SQP algorithms to solve benchmark nonlinear problems in CUTEst test set [7] and solve constrained logistic regression with datasets from LIBSVM [6].

Original: The considered three algorithms are Algorithm 1 (the proposed algorithm), Algorithm 2: [4] with the ℓ 1 penalized merit function, and Algorithm 3: adaptive version of Algorithm 2.
Revised: The considered three algorithms are Algorithm 1 (the proposed algorithm), Algorithm 2: [4] with the ℓ 1 penalized merit function, and Algorithm 3: adaptive version of Algorithm 2.

Original: We use two randomized iterative solvers and one deterministic iterative solver for (2.3): Gaussian vector sketch [8, Section 3.2], Randomized Kaczmarz [8, Section 3.3], and GMRES [15].
Revised: We use two randomized iterative solvers and one deterministic iterative solver for (2.3): Gaussian vector sketch [8, Section 3.2], Randomized Kaczmarz [8, Section 3.3], and GMRES [15].

Original: We evaluate each algorithm with the following three criteria: (1) the KKT residual ( ∥∇L k ∥ ) , (2) the number of gradient and Jacobian evaluations, and (3) the number of objective and constraints evaluations.
Revised: We evaluate each algorithm with the following three criteria: (1) the KKT residual ( ∥∇L k ∥ ) , (2) the number of gradient and Jacobian evaluations, and (3) the number of objective and constraints evaluations.

Original: We first present the comparison between Algorithms 1 and 3 on CUTEst set in Figure 1.
Revised: We first present the comparison between Algorithms 1 and 3 on CUTEst set in Figure 1.

Original: From Figure 1, we observe that Algorithm 1 outperforms Algorithm 3 in terms of the KKT residual and number of objective and constraints evaluations.
Revised: From Figure 1, we observe that Algorithm 1 outperforms Algorithm 3 in terms of the KKT residual and number of objective and constraints evaluations.

Original: This result is expected as Algorithmuses tighter bounds on the residuals of the iterative solver to guarantee fast local convergence.
Revised: This result is expected as Algorithmuses tighter bounds on the residuals of the iterative solver to guarantee fast local convergence.

Original: This results in steeper decrease in the merit function at each iteration and fewer number of outer iterations.
Revised: This results in steeper decrease in the merit function at each iteration and fewer number of outer iterations.

Original: However, as we mentioned earlier, smaller number of outer iterations yields possible increase in the number of inner iterations required to satisfy (2.7).
Revised: However, as we mentioned earlier, smaller number of outer iterations yields possible increase in the number of inner iterations required to satisfy (2.7).

Original: We can reduce this cost by applying IRS, which substantially saves the computational complexities by projecting (2.3) into a smaller space for each inner iteration.
Revised: We can reduce this cost by applying IRS, which substantially saves the computational complexities by projecting (2.3) into a smaller space for each inner iteration.

Original: On the one hand, we see Algorithm 3 shows slightly lower number of gradient and Jacobian evaluations than Algorithm 1.
Revised: On the one hand, we see Algorithm 3 shows slightly lower number of gradient and Jacobian evaluations than Algorithm 1.

Original: This is because for each iteration, Algorithm 1 finds a stepsize α k to satisfy the Armijo condition (2.10), and L η ( x k + α k ˜∆ x k , λ k + α k ˜∆ λ k ) in (2.10) requires the gradient and Jacobian to be evaluated at each new trial point.
Revised: This is because for each iteration, Algorithm 1 finds a stepsize α k to satisfy the Armijo condition (2.10), and L η ( x k + α k ˜∆ x k , λ k + α k ˜∆ λ k ) in (2.10) requires the gradient and Jacobian to be evaluated at each new trial point.

Original: For Algorithm 3, however, the gradient and Jacobian are not involved in the evaluations of the ℓ 1 penalized merit function at new trial points.
Revised: For Algorithm 3, however, the gradient and Jacobian are not involved in the evaluations of the ℓ 1 penalized merit function at new trial points.

Original: Further comparisons between Algorithms 2 and 3, and experiments on LIBSVM datasets are in
Revised: Further comparisons between Algorithms 2 and 3, and experiments on LIBSVM datasets are in

Original: Appendix D due to the limitation of space.
Revised: Appendix D due to the limitation of space.

Original: The code is available at: https://github.com/
Revised: The code is available at: https://github.com/

Original: IlgeeHong/Randomized-SQP .
Revised: IlgeeHong/Randomized-SQP .

Original: Appendix A.
Revised: Appendix A.

Original: Lemma 3 (Upper bound on Hessian of Lagrangian) Under Assumption 1, 2, 3, for any outer iteration k , there exists a uniform constant Υ H > 0 , independent of k , such that for any outer iteration k , ∥ H k ∥ ≤ Υ H .
Revised: Lemma 3 (Upper bound on Hessian of Lagrangian) Under Assumption 1, 2, 3, for any outer iteration k , there exists a uniform constant Υ H > 0 , independent of k , such that for any outer iteration k , ∥ H k ∥ ≤ Υ H .

Original: Lemma 4 (Upper bound on Newton matrix)
Revised: Lemma 4 (Upper bound on Newton matrix)

Original: Under Assumption 1, 2, for any outer iteration k , there exists a uniform constant Υ N > 0 , independent of k , such that for any outer iteration k , ∥ Γ k ∥ ≤ Υ N .
Revised: Under Assumption 1, 2, for any outer iteration k , there exists a uniform constant Υ N > 0 , independent of k , such that for any outer iteration k , ∥ Γ k ∥ ≤ Υ N .

Original: Lemma 5 (Boundedness on Jacobian of constraints)
Revised: Lemma 5 (Boundedness on Jacobian of constraints)

Original: Under Assumption 2, there exist constants κ G , ξ G > 0 such that for any outer iteration k , ξ G I ⪯ G k G Tk ⪯ κ G I .
Revised: Under Assumption 2, there exist constants κ G , ξ G > 0 such that for any outer iteration k , ξ G I ⪯ G k G Tk ⪯ κ G I .

Original: Lemma 6 (Upper bound on Newton matrix inverse)
Revised: Lemma 6 (Upper bound on Newton matrix inverse)

Original: Under Assumption 2, for any outer iteration k , we let Ψ k
Revised: Under Assumption 2, for any outer iteration k , we let Ψ k

Original: = 7( ∥ B k ∥ 2 ∨ 1) ξ B ( σ 1 ,k ∧ 1) where σ 1 ,k is the smallest eigenvalue of G k G Tk .
Revised: = 7( ∥ B k ∥ 2 ∨ 1) ξ B ( σ 1 ,k ∧ 1) where σ 1 ,k is the smallest eigenvalue of G k G Tk .

Original: Then for any outer iteration k , (cid:13)(cid:13)
Revised: Then for any outer iteration k , (cid:13)(cid:13)

Original: Γ − 1 k (cid:13)(cid:13) ≤ Ψ k .
Revised: Γ − 1 k (cid:13)(cid:13) ≤ Ψ k .

Original: For later usage, we further define Ψ = sup k ≥ 

Revised: For later usage, we further define Ψ = sup k ≥

Original: { Ψ k } and Υ = sup k ≥ 

Revised: { Ψ k } and Υ = sup k ≥

Original: Υ k } .
Revised: Υ k } .

Original: Lemma 7 Let Q k,j be a random matrix with orthonormal columns that form a basis of row ( S Tk,j Γ k ) .
Revised: Lemma 7 Let Q k,j be a random matrix with orthonormal columns that form a basis of row ( S Tk,j Γ k ) .

Original: Let { j l } l ≥ 0 be a subsequence of the inner iteration where j 0 = 

Revised: Let { j l } l ≥ 0 be a subsequence of the inner iteration where j 0 =

Original: and j l be the l -th iteration such that
Revised: and j l be the l -th iteration such that

Original: [Equation]
Revised: [Equation]

Original: Let L be any given positive integer.
Revised: Let L be any given positive integer.

Original: Under Assumption 2, 3, for any outer iteration k , conditioned on the event that the algorithm reaches ( x k , λ k ) , the event
Revised: Under Assumption 2, 3, for any outer iteration k , conditioned on the event that the algorithm reaches ( x k , λ k ) , the event

Original: [Equation]
Revised: [Equation]

Original: (A.1) happens with probability 1.
Revised: (A.1) happens with probability 1.

Original: Lemma 8 (Subsequence of error linearly decays)
Revised: Lemma 8 (Subsequence of error linearly decays)

Original: Under Assumption 2, 3, for any outer iteration k and for any positive integer L , conditioned on the event that the algorithm reaches ( x k , λ k ) and
Revised: Under Assumption 2, 3, for any outer iteration k and for any positive integer L , conditioned on the event that the algorithm reaches ( x k , λ k ) and

Original: A k in (A.1) , there exists a sequence of random variables { γ k,l } Ll =1 where γ k,l i.i.d.
Revised: A k in (A.1) , there exists a sequence of random variables { γ k,l } Ll =1 where γ k,l i.i.d.

Original: ∼ γ k ∈ [0 , 1) such that, for any l ≤ L ,
Revised: ∼ γ k ∈ [0 , 1) such that, for any l ≤ L ,

Original: [Equation]
Revised: [Equation]

Original: Lemma 9 (Error of inexact solution)
Revised: Lemma 9 (Error of inexact solution)

Original: For any δ k ∈ (0 , 1) , let J k be the inner iteration such that
Revised: For any δ k ∈ (0 , 1) , let J k be the inner iteration such that

Original: [Equation]
Revised: [Equation]

Original: Under Assumption 1, 2, 3, for any outer iteration k , conditioned on the event that the algorithm reaches ( x k , λ k ) and A k in (A.1) , the event
Revised: Under Assumption 1, 2, 3, for any outer iteration k , conditioned on the event that the algorithm reaches ( x k , λ k ) and A k in (A.1) , the event

Original: [Equation]
Revised: [Equation]

Original: happens with probability 1.
Revised: happens with probability 1.

Original: Moreover, conditioned on the event A k ∩ B k , if we let ( ˜∆ x k , ˜∆ λ k )
Revised: Moreover, conditioned on the event A k ∩ B k , if we let ( ˜∆ x k , ˜∆ λ k )

Original: = ( ˜∆ x k,J k , ˜∆ λ k,J k ) , then
Revised: = ( ˜∆ x k,J k , ˜∆ λ k,J k ) , then

Original: [Equation]
Revised: [Equation]

Original: Lemma 10 (Descent direction of inexact step) Let ( ˜∆ x k , ˜∆ λ k ) be the inexact solution that satisfies (2.7) .
Revised: Lemma 10 (Descent direction of inexact step) Let ( ˜∆ x k , ˜∆ λ k ) be the inexact solution that satisfies (2.7) .

Original: Under Assumption 1, 2, 3, for any outer iteration k , conditioned on the event that the algorithm reaches ( x k , λ k ) and A k ∩ B k in (A.1) and (A.2) , if
Revised: Under Assumption 1, 2, 3, for any outer iteration k , conditioned on the event that the algorithm reaches ( x k , λ k ) and A k ∩ B k in (A.1) and (A.2) , if

Original: [Equation]
Revised: [Equation]

Original: then we have
Revised: then we have

Original: [Equation]
Revised: [Equation]

Original: Lemma 11 (Stability of adaptive parameters)
Revised: Lemma 11 (Stability of adaptive parameters)

Original: Under Assumption 1, 2, 3, after sufficiently large outer iteration k , all adaptive parameters ( η 1 , η 2 , δ ) are stabilized almost surely.
Revised: Under Assumption 1, 2, 3, after sufficiently large outer iteration k , all adaptive parameters ( η 1 , η 2 , δ ) are stabilized almost surely.

Original: Lemma 12 (Armijo condition)
Revised: Lemma 12 (Armijo condition)

Original: Under Assumption 1, 2, 3, for any outer iteration k , conditioned on the event that the algorithm reaches ( x k , λ k ) and A k ∩ B k in (A.1) and (A.2) , the Armijo condition (2.10) is satisfied.
Revised: Under Assumption 1, 2, 3, for any outer iteration k , conditioned on the event that the algorithm reaches ( x k , λ k ) and A k ∩ B k in (A.1) and (A.2) , the Armijo condition (2.10) is satisfied.

Original: Moreover, there exists a uniform constant α min > 0 , independent of k , such that for any k , 0 < α min ≤ α k .
Revised: Moreover, there exists a uniform constant α min > 0 , independent of k , such that for any k , 0 < α min ≤ α k .

Original: Lemma 13 (Boundedness of dual variable)
Revised: Lemma 13 (Boundedness of dual variable)

Original: Under Assumption 1, 2, 3, almost surely, { λ k } k ≥produced by Algorithm 1 is bounded.
Revised: Under Assumption 1, 2, 3, almost surely, { λ k } k ≥produced by Algorithm 1 is bounded.

Original: Appendix B. Proof of Lemma and Theorem B.1. Proof of Lemma
Revised: Appendix B. Proof of Lemma and Theorem B.1. Proof of Lemma

Original: Proof Under Assumption 1, 2, 3, Lemma 13 shows { λ k } k ≥ 0 is bounded.
Revised: Proof Under Assumption 1, 2, 3, Lemma 13 shows { λ k } k ≥ 0 is bounded.

Original: Using Assumption 1, we have for any outer iteration k , ∇ 2 f k , ∇ 2 c
Revised: Using Assumption 1, we have for any outer iteration k , ∇ 2 f k , ∇ 2 c

Original: i,k are all bounded.
Revised: i,k are all bounded.

Original: Then we get
Revised: Then we get

Original: [Equation]
Revised: [Equation]

Original: [Table] 
Revised: [Table] [Equation] [Table] [Equation]

Original: [Equation] [Table] 
Revised: [Table] [Equation]

Original: [Equation] [Table] [Equation] [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: This ends proof of Lemma 3.
Revised: This ends proof of Lemma 3.

Original: B.2. Proof of Lemma
Revised: B.2. Proof of Lemma

Original: Proof Assumption 1 implies that for any outer iteration k , there exists a uniform constant Υ G > 0 , independent of k , such that for any outer iteration k , ∥ G k ∥ ≤ Υ G .
Revised: Proof Assumption 1 implies that for any outer iteration k , there exists a uniform constant Υ G > 0 , independent of k , such that for any outer iteration k , ∥ G k ∥ ≤ Υ G .

Original: Using this fact together with Assumption 2, we get
Revised: Using this fact together with Assumption 2, we get

Original: [Equation]
Revised: [Equation]

Original: This ends proof of Lemma 4.
Revised: This ends proof of Lemma 4.

Original: B.3. Proof of Lemma
Revised: B.3. Proof of Lemma

Original: Proof Assumption 2 implies that for any outer iteration k , G k G Tk is positive definite.
Revised: Proof Assumption 2 implies that for any outer iteration k , G k G Tk is positive definite.

Original: For any outer iteration k , let σ m,k ≥ · · · ≥ σ 1 ,k > 0 be the eigenvalues of G k G Tk .
Revised: For any outer iteration k , let σ m,k ≥ · · · ≥ σ 1 ,k > 0 be the eigenvalues of G k G Tk .

Original: Then we can show σ 1 ,k I ⪯ G k G Tk ⪯ σ m,k I .
Revised: Then we can show σ 1 ,k I ⪯ G k G Tk ⪯ σ m,k I .

Original: If we let ξ G = inf k ≥ 0 { σ 1 ,k } , and κ G = sup k ≥ 0 { σ m,k } then for any outer iteration k , ξ G I ⪯ G k G Tk ⪯ κ G I .
Revised: If we let ξ G = inf k ≥ 0 { σ 1 ,k } , and κ G = sup k ≥ 0 { σ m,k } then for any outer iteration k , ξ G I ⪯ G k G Tk ⪯ κ G I .

Original: This ends proof of Lemma 5.
Revised: This ends proof of Lemma 5.

Original: B.4. Proof of Lemma
Revised: B.4. Proof of Lemma

Original: Proof For any outer iteration k , let Z k be a matrix which has orthonormal columns spanning the null space of G k .
Revised: Proof For any outer iteration k , let Z k be a matrix which has orthonormal columns spanning the null space of G k .

Original: Using Assumption 2, we have Z Tk B k Z k ⪰ ξ B I and G Tk ( G k G Tk ) − 1 G k + Z k Z Tk
Revised: Using Assumption 2, we have Z Tk B k Z k ⪰ ξ B I and G Tk ( G k G Tk ) − 1 G k + Z k Z Tk

Original: = I .
Revised: = I .

Original: Appendix C.1.
Revised: Appendix C.1.

Original: in [11] and [12] implies that
Revised: in [11] and [12] implies that

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Taking ℓ 2 norm on both sides yields,
Revised: Taking ℓ 2 norm on both sides yields,

Original: [Equation]
Revised: [Equation]

Original: Let ξ B ≤ 1 and assume σ 1 ,k ≤ 1 ≤ ∥
Revised: Let ξ B ≤ 1 and assume σ 1 ,k ≤ 1 ≤ ∥

Original: B k ∥ .
Revised: B k ∥ .

Original: Then we get
Revised: Then we get

Original: [Equation]
Revised: [Equation]

Original: Since we assume σ 1 ,k ≤ 1 ≤ ∥ B k ∥
Revised: Since we assume σ 1 ,k ≤ 1 ≤ ∥ B k ∥

Original: [Equation]
Revised: [Equation]

Original: it follows that
Revised: it follows that

Original: [Equation]
Revised: [Equation]

Original: This ends proof of Lemma 6.
Revised: This ends proof of Lemma 6.

Original: B
Revised: B

Original: [Equation]
Revised: [Equation]

Original: 5. Proof of Lemma
Revised: 5. Proof of Lemma

Original: Proof Let k ≥ 

Revised: Proof Let k ≥

Original: and we suppose the algorithm reaches ( x k , λ k ) .
Revised: and we suppose the algorithm reaches ( x k , λ k ) .

Original: Using (2.3) and (2.4), we rewrite the updating rule of IRS (2.5) as
Revised: Using (2.3) and (2.4), we rewrite the updating rule of IRS (2.5) as

Original: [Equation]
Revised: [Equation]

Original: If we let e k,j = (cid:18) ˜∆ x k,j − ∆ x k ˜∆ λ k,j − ∆ λ k
Revised: If we let e k,j = (cid:18) ˜∆ x k,j − ∆ x k ˜∆ λ k,j − ∆ λ k

Original: [Equation]
Revised: [Equation]

Original: , then the above display can be rewritten as
Revised: , then the above display can be rewritten as

Original: [Equation]
Revised: [Equation]

Original: Using the fact that W k,j Γ k = Γ Tk S k,j +1 ( S Tk,
Revised: Using the fact that W k,j Γ k = Γ Tk S k,j +1 ( S Tk,

Original: j +1 Γ k Γ
Revised: j +1 Γ k Γ

Original: Tk S k,j +1 )
Revised: Tk S k,j +1 )

Original: − 1 S Tk,j +1 Γ k forms an orthogonal projection onto row ( S Tk,j +1 Γ k ) , (B.1) can be simplified as
Revised: − 1 S Tk,j +1 Γ k forms an orthogonal projection onto row ( S Tk,j +1 Γ k ) , (B.1) can be simplified as

Original: [Equation]
Revised: [Equation]

Original: Let j 0 = 

Revised: Let j 0 =

Original: and j l be the l -th iteration such that
Revised: and j l be the l -th iteration such that

Original: [Equation]
Revised: [Equation]

Original: otherwise let j l be infinite.
Revised: otherwise let j l be infinite.

Original: Since Γ k is invertible, Assumption 3 implies P ( S T Γ k z ̸ = 0) > 0 for any z ∈ R n + m \{ 0 } .
Revised: Since Γ k is invertible, Assumption 3 implies P ( S T Γ k z ̸ = 0) > 0 for any z ∈ R n + m \{ 0 } .

Original: Given the relationship between row ( S T Γ k ) and Q k , we further get P ( Q Tk z ̸ = 0) >for any z ∈ R n + m \{ 0 } .
Revised: Given the relationship between row ( S T Γ k ) and Q k , we further get P ( Q Tk z ̸ = 0) >for any z ∈ R n + m \{ 0 } .

Original: We denote the lower bound of this probability as π k ∈ (0 , 1] .
Revised: We denote the lower bound of this probability as π k ∈ (0 , 1] .

Original: Since Q k,j i.i.d.
Revised: Since Q k,j i.i.d.

Original: ∼
Revised: ∼

Original: Q k , conditioned on the event { j l − 1 < ∞} , the probability that (cid:80) t +1 i =0 col (cid:0) Q k,j l − 1 + i (cid:1) grows in dimension relative to (cid:80) ti =

Revised: Q k , conditioned on the event { j l − 1 < ∞} , the probability that (cid:80) t +1 i =0 col (cid:0) Q k,j l − 1 + i (cid:1) grows in dimension relative to (cid:80) ti =

Original: col (cid:0) Q k,j l − 1 + i (cid:1) , when dim (cid:16)(cid:80) ti =

Revised: col (cid:0) Q k,j l − 1 + i (cid:1) , when dim (cid:16)(cid:80) ti =

Original: col (cid:0) Q k,j l
Revised: col (cid:0) Q k,j l

Original: [Equation]
Revised: [Equation]

Original: 1 + i (cid:1)(cid:17) < n + m is at least π k .
Revised: 1 + i (cid:1)(cid:17) < n + m is at least π k .

Original: As a result, conditioned on the event { j l − 1 < ∞} , the probability that the event
Revised: As a result, conditioned on the event { j l − 1 < ∞} , the probability that the event

Original: [Equation]
Revised: [Equation]

Original: happens n + m times in N iterations with N ≥ n + m is dominated by a negative binomial distribution.
Revised: happens n + m times in N iterations with N ≥ n + m is dominated by a negative binomial distribution.

Original: Thus, for N ≥ n
Revised: Thus, for N ≥ n

Original: [Equation]
Revised: [Equation]

Original: m
Revised: m

Original: [Equation]
Revised: [Equation]

Original: P
Revised: P

Original: [Equation]
Revised: [Equation]

Original: N
Revised: N

Original: [Equation]
Revised: [Equation]

Original: A
Revised: A

Original: Taking N → ∞ , we get for any l ∈ N ,
Revised: Taking N → ∞ , we get for any l ∈ N ,

Original: [Equation]
Revised: [Equation]

Original: Therefore, for any l ∈ N , P ( j l < ∞| j l − 1 < ∞ ) = 1 .
Revised: Therefore, for any l ∈ N , P ( j l < ∞| j l − 1 < ∞ ) = 1 .

Original: Let L be given positive integer.
Revised: Let L be given positive integer.

Original: Then for any l ≤ L ,
Revised: Then for any l ≤ L ,

Original: P
Revised: P

Original: [Equation]
Revised: [Equation]

Original: This ends proof of Lemma 7.
Revised: This ends proof of Lemma 7.

Original: B.6. Proof of Lemma
Revised: B.6. Proof of Lemma

Original: Proof Let k ≥ 

Revised: Proof Let k ≥

Original: and we suppose the algorithm reaches ( x k , λ k ) and the event
Revised: and we suppose the algorithm reaches ( x k , λ k ) and the event

Original: A k happens.
Revised: A k happens.

Original: Let L be given positive integer.
Revised: Let L be given positive integer.

Original: We denote q k,j,h be the h -th column of Q k,j .
Revised: We denote q k,j,h be the h -th column of Q k,j .

Original: Using (B.2), we have for any l ≤ L ,
Revised: Using (B.2), we have for any l ≤ L ,

Original: [Equation]
Revised: [Equation]

Original: Taking ℓ 2 norm on both sides yields
Revised: Taking ℓ 2 norm on both sides yields

Original: [Equation]
Revised: [Equation]

Original: Let F k,l denote all matrices F k,l where the columns of F k,l are the vectors { f k,l, 1 , . . .
Revised: Let F k,l denote all matrices F k,l where the columns of F k,l are the vectors { f k,l, 1 , . . .

Original: , f k,l,n + m } ⊂ { q k,j l − 1 +1 , 1 , . . .
Revised: , f k,l,n + m } ⊂ { q k,j l − 1 +1 , 1 , . . .

Original: , q k,j l ,d } that are a maximal linearly independent subset.
Revised: , q k,j l ,d } that are a maximal linearly independent subset.

Original: Theorem 4 in [13] implies that
Revised: Theorem 4 in [13] implies that

Original: [Equation]
Revised: [Equation]

Original: For any l ≤ L , define
Revised: For any l ≤ L , define

Original: [Equation]
Revised: [Equation]

Original: Then we have for any l ≤ L ,
Revised: Then we have for any l ≤ L ,

Original: [Equation]
Revised: [Equation]

Original: Using the fact that F Tk,l F k,l is positive definite and Hadamard’s inequality, we have { γ k,l } l ≤ L ⊂ [0 , 1) .
Revised: Using the fact that F Tk,l F k,l is positive definite and Hadamard’s inequality, we have { γ k,l } l ≤ L ⊂ [0 , 1) .

Original: Let
Revised: Let

Original: Q k,l = { Q k,j l − 1 +1 , . . .
Revised: Q k,l = { Q k,j l − 1 +1 , . . .

Original: , Q k,j l } .
Revised: , Q k,j l } .

Original: Using Assumption 3, we get Q k, 1 , . . .
Revised: Using Assumption 3, we get Q k, 1 , . . .

Original: , Q k,L i.i.d ∼ Q k , hence, γ k, 1 , . .
Revised: , Q k,L i.i.d ∼ Q k , hence, γ k, 1 , . .

Original: . , γ k,L i.i.d ∼ γ k .
Revised: . , γ k,L i.i.d ∼ γ k .

Original: This ends proof of Lemma 8.
Revised: This ends proof of Lemma 8.

Original: B.7. Proof of Lemma
Revised: B.7. Proof of Lemma

Original: Proof Let k ≥ 

Revised: Proof Let k ≥

Original: and we suppose the algorithm reaches ( x k , λ k ) and the event
Revised: and we suppose the algorithm reaches ( x k , λ k ) and the event

Original: A k happens.
Revised: A k happens.

Original: Using
Revised: Using

Original: Lemma 8, we have P ( γ k = 1) = 0 , hence, there exists τ k ∈ (0 , 1) such that P ( γ k ≤ τ k ) > 0 .
Revised: Lemma 8, we have P ( γ k = 1) = 0 , hence, there exists τ k ∈ (0 , 1) such that P ( γ k ≤ τ k ) > 0 .

Original: We denote the lower bound of P ( γ k ≤ τ k ) by π k ∈ (0 , 1] .
Revised: We denote the lower bound of P ( γ k ≤ τ k ) by π k ∈ (0 , 1] .

Original: Let ¯ N be the smallest positive integer such that ¯ N ≥ log ( δ k / ∥
Revised: Let ¯ N be the smallest positive integer such that ¯ N ≥ log ( δ k / ∥

Original: Γ k ∥ 2 Ψ 2 k ) /log
Revised: Γ k ∥ 2 Ψ 2 k ) /log

Original: ( τ k ) + 1 .
Revised: ( τ k ) + 1 .

Original: Then we have τ ¯ Nk ≤ δ
Revised: Then we have τ ¯ Nk ≤ δ

Original: [Equation]
Revised: [Equation]

Original: ∥
Revised: ∥

Original: [Equation]
Revised: [Equation]

Original: ∥
Revised: ∥

Original: [Equation]
Revised: [Equation]

Original: Now we consider the procedure where for each iteration l , we generate γ k,l from a distribution of γ k independently.
Revised: Now we consider the procedure where for each iteration l , we generate γ k,l from a distribution of γ k independently.

Original: Let L k be the iteration such that
Revised: Let L k be the iteration such that

Original: [Equation]
Revised: [Equation]

Original: otherwise let L k be infinite.
Revised: otherwise let L k be infinite.

Original: Since for any l ∈ N , P ( γ k,l ≤ τ k ) ≥ π k and γ k,l i.i
Revised: Since for any l ∈ N , P ( γ k,l ≤ τ k ) ≥ π k and γ k,l i.i

Original: [Equation]
Revised: [Equation]

Original: d. ∼ γ k , the probability that the event { γ k,l ≤ τ k } happens ¯ N times in N iterations with N ≥ ¯ N is dominated by a negative binomial distribuiton.
Revised: d. ∼ γ k , the probability that the event { γ k,l ≤ τ k } happens ¯ N times in N iterations with N ≥ ¯ N is dominated by a negative binomial distribuiton.

Original: Thus,
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: Taking N → ∞
Revised: Taking N → ∞

Original: [Equation]
Revised: [Equation]

Original: we get
Revised: we get

Original: [Equation]
Revised: [Equation]

Original: Therefore, L k is finite with probability 1.
Revised: Therefore, L k is finite with probability 1.

Original: Now letting L = L k and applying Lemma
Revised: Now letting L = L k and applying Lemma

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Using this expression together with (2.3), (2.4)
Revised: Using this expression together with (2.3), (2.4)

Original: [Equation]
Revised: [Equation]

Original: and Lemma 6 which says (cid:13)(cid:13)
Revised: and Lemma 6 which says (cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: ≤ Ψ k , we get (cid:13) (cid:13) (cid:13) r k
Revised: ≤ Ψ k , we get (cid:13) (cid:13) (cid:13) r k

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: L
Revised: L

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: Using (B.3) we have
Revised: Using (B.3) we have

Original: [Equation]
Revised: [Equation]

Original: (cid:27)
Revised: (cid:27)

Original: Finally, if we let J k =
Revised: Finally, if we let J k =

Original: j L k , then we obtain
Revised: j L k , then we obtain

Original: P (cid:18) there exists finite J k such that ∥ r k,J k ∥ ≤ δ k ∥∇L k ∥ ∥
Revised: P (cid:18) there exists finite J k such that ∥ r k,J k ∥ ≤ δ k ∥∇L k ∥ ∥

Original: Γ k ∥ Ψ k (cid:12)(cid:12)(cid:12)(cid:12) ( x k , λ k ) , A k (cid:19) = 1 .
Revised: Γ k ∥ Ψ k (cid:12)(cid:12)(cid:12)(cid:12) ( x k , λ k ) , A k (cid:19) = 1 .

Original: Now conditioned on the event A k ∩ B k , we get
Revised: Now conditioned on the event A k ∩ B k , we get

Original: [Equation]
Revised: [Equation]

Original: ⇒
Revised: ⇒

Original: [Equation]
Revised: [Equation]

Original: ∥∇L
Revised: ∥∇L

Original: [Equation]
Revised: [Equation]

Original: ⇒
Revised: ⇒

Original: [Equation]
Revised: [Equation]

Original: ˜∆
Revised: ˜∆

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: This ends proof of Lemma
Revised: This ends proof of Lemma

Original: [Table]
Revised: [Table]

Original: B.8. Proof of Lemma
Revised: B.8. Proof of Lemma

Original: Proof Let k ≥ 

Revised: Proof Let k ≥

Original: and we suppose the algorithm reaches ( x k , λ k ) and the event
Revised: and we suppose the algorithm reaches ( x k , λ k ) and the event

Original: A k ∩ B k happens.
Revised: A k ∩ B k happens.

Original: We start from dividing (cid:18) ∇
Revised: We start from dividing (cid:18) ∇

Original: [Equation]
Revised: [Equation]

Original: L
Revised: L

Original: [Equation]
Revised: [Equation]

Original: (cid:19) into two terms as follows:
Revised: (cid:19) into two terms as follows:

Original: [Equation]
Revised: [Equation]

Original: First
Revised: First

Original: [Equation]
Revised: [Equation]

Original: we develop the first term and obtain (cid:18) ∇ x L k η
Revised: we develop the first term and obtain (cid:18) ∇ x L k η

Original: [Equation]
Revised: [Equation]

Original: Using (2.2) we have G k ∆ x = − c k and G Tk ∆ λ k = − ( B k ∆ x k + ∇ x L k ) .
Revised: Using (2.2) we have G k ∆ x = − c k and G Tk ∆ λ k = − ( B k ∆ x k + ∇ x L k ) .

Original: Using this expression together with the above display
Revised: Using this expression together with the above display

Original: [Equation]
Revised: [Equation]

Original: we obtain
Revised: we obtain

Original: [Equation]
Revised: [Equation]

Original: Taking the forth and fifth terms from the above display and using the expression ∇ x L k = − (cid:0) B k ∆ x k + G Tk ∆ λ k (cid:1)
Revised: Taking the forth and fifth terms from the above display and using the expression ∇ x L k = − (cid:0) B k ∆ x k + G Tk ∆ λ k (cid:1)

Original: [Equation]
Revised: [Equation]

Original: we obtain
Revised: we obtain

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: A
Revised: A

Original: Combining the above two displays we get
Revised: Combining the above two displays we get

Original: [Equation]
Revised: [Equation]

Original: Assuming η 1 ,k ≥ η 2 ,k at the moment and using Cauchy-Schwarz inequality
Revised: Assuming η 1 ,k ≥ η 2 ,k at the moment and using Cauchy-Schwarz inequality

Original: [Equation]
Revised: [Equation]

Original: we get (cid:18) ∇ x L k η
Revised: we get (cid:18) ∇ x L k η

Original: [Equation]
Revised: [Equation]

Original: ∆ x k
Revised: ∆ x k

Original: [Equation]
Revised: [Equation]

Original: Now we apply Young’s inequality for the last two terms.
Revised: Now we apply Young’s inequality for the last two terms.

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: Using the above expression
Revised: Using the above expression

Original: [Equation]
Revised: [Equation]

Original: we get
Revised: we get

Original: [Equation]
Revised: [Equation]

Original: ∆ λ k (cid:13)(cid:13) ∥
Revised: ∆ λ k (cid:13)(cid:13) ∥

Original: ∆ x k ∥ ≤ η 2 ,k 4 (cid:13)(cid:13)
Revised: ∆ x k ∥ ≤ η 2 ,k 4 (cid:13)(cid:13)

Original: G Tk ∆ λ k
Revised: G Tk ∆ λ k

Original: (cid:13)(cid:13) 

Revised: (cid:13)(cid:13)

Original: + η 2 ,k Υ 2 ∥
Revised: + η 2 ,k Υ 2 ∥

Original: ∆ x k ∥and
Revised: ∆ x k ∥and

Original: [Equation]
Revised: [Equation]

Original: Using Lemma
Revised: Using Lemma

Original: [Equation]
Revised: [Equation]

Original: we obtain
Revised: we obtain

Original: [Equation]
Revised: [Equation]

Original: In order to bound the second and fourth terms from the above display, we decompose ∆ x k as ∆ x k = ∆ u k + ∆ v k where ∆ u k ∈ Null ( G k ) and ∆ v k ∈ Image ( G Tk ) .
Revised: In order to bound the second and fourth terms from the above display, we decompose ∆ x k as ∆ x k = ∆ u k + ∆ v k where ∆ u k ∈ Null ( G k ) and ∆ v k ∈ Image ( G Tk ) .

Original: Then we have ∥ ∆ x k ∥ 2 = ∥
Revised: Then we have ∥ ∆ x k ∥ 2 = ∥

Original: ∆ u k ∥ 2 + ∥ ∆ v k ∥ 2 and ∆ v k = G Tk ∆¯ v k for some ∆¯ v k .
Revised: ∆ u k ∥ 2 + ∥ ∆ v k ∥ 2 and ∆ v k = G Tk ∆¯ v k for some ∆¯ v k .

Original: Using Lemma 5, we get ∥ ∆ v k ∥ 2 = ∥
Revised: Using Lemma 5, we get ∥ ∆ v k ∥ 2 = ∥

Original: G Tk ∆¯ v k ∥ 2 ≤ κ G ∥ ∆¯ v k ∥ 2 and further obtain
Revised: G Tk ∆¯ v k ∥ 2 ≤ κ G ∥ ∆¯ v k ∥ 2 and further obtain

Original: [Equation]
Revised: [Equation]

Original: Using the above expressions and Cauchy-Schwarz inequality, and assuming η 1 ,k ≥ 16 / ( η 2 ,k ξ G ) at the momonet, we get
Revised: Using the above expressions and Cauchy-Schwarz inequality, and assuming η 1 ,k ≥ 16 / ( η 2 ,k ξ G ) at the momonet, we get

Original: [Equation]
Revised: [Equation]

Original: Now we apply Young’s inequality for the second term.
Revised: Now we apply Young’s inequality for the second term.

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: Using the above expression
Revised: Using the above expression

Original: [Equation]
Revised: [Equation]

Original: we get 2Υ
Revised: we get 2Υ

Original: [Equation]
Revised: [Equation]

Original: This leads to
Revised: This leads to

Original: [Equation]
Revised: [Equation]

Original: Plugging the above inequality back into (B.5), we get (cid:18) ∇ x L k η ∇
Revised: Plugging the above inequality back into (B.5), we get (cid:18) ∇ x L k η ∇

Original: [Equation]
Revised: [Equation]

Original: In order to make the upper bound negative
Revised: In order to make the upper bound negative

Original: [Equation]
Revised: [Equation]

Original: we let
Revised: we let

Original: [Equation]
Revised: [Equation]

Original: Furthermore, without loss of generality
Revised: Furthermore, without loss of generality

Original: [Equation]
Revised: [Equation]

Original: we assume κ G ∧ Υ / 2 ≥ 1 ≥ ξ B ∨ ξ G .
Revised: we assume κ G ∧ Υ / 2 ≥ 1 ≥ ξ B ∨ ξ G .

Original: Using this assumption together with (B.6), we obtain
Revised: Using this assumption together with (B.6), we obtain

Original: [Equation]
Revised: [Equation]

Original: A
Revised: A

Original: Using (B.6) togther with the above inequality, we get
Revised: Using (B.6) togther with the above inequality, we get

Original: [Equation]
Revised: [Equation]

Original: In order to make the upper bound negative
Revised: In order to make the upper bound negative

Original: [Equation]
Revised: [Equation]

Original: we let
Revised: we let

Original: [Equation]
Revised: [Equation]

Original: Note that (B.6) and (B.7) imply η 1 ,k ≥ η 2 ,k and η 1 ,k ≥ 16 / ( η 2 ,k ξ G ) , hence, justify our previous assumption.
Revised: Note that (B.6) and (B.7) imply η 1 ,k ≥ η 2 ,k and η 1 ,k ≥ 16 / ( η 2 ,k ξ G ) , hence, justify our previous assumption.

Original: Using (B.6) and (B.7)
Revised: Using (B.6) and (B.7)

Original: [Equation]
Revised: [Equation]

Original: we finally have
Revised: we finally have

Original: [Equation]
Revised: [Equation]

Original: Now we develop the second term of (B.4).
Revised: Now we develop the second term of (B.4).

Original: Using Cauchy-Schwarz inequality and (B.6), we
Revised: Using Cauchy-Schwarz inequality and (B.6), we

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Furthermore, without loss of generality, we assume κ G ∧ Υ / 2 ≥ 1 ≥ ξ B ∨ ξ G .
Revised: Furthermore, without loss of generality, we assume κ G ∧ Υ / 2 ≥ 1 ≥ ξ B ∨ ξ G .

Original: Using (B.6) and (B.7), we get η 1 ,k ≥ (17 κ G ) / ( η 2 ,k ξ 2 G ) ≥ 17 / ( η 2 ,k ξ G ) ≥ (17 × 12Υ 2 ) / ( ξ B ξ G ) .
Revised: Using (B.6) and (B.7), we get η 1 ,k ≥ (17 κ G ) / ( η 2 ,k ξ 2 G ) ≥ 17 / ( η 2 ,k ξ G ) ≥ (17 × 12Υ 2 ) / ( ξ B ξ G ) .

Original: Then we have 19 / 6 ≤ η 1 ,k Υ and further obtain
Revised: Then we have 19 / 6 ≤ η 1 ,k Υ and further obtain

Original: [Equation]
Revised: [Equation]

Original: Using this inequality, we finally have
Revised: Using this inequality, we finally have

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Now plugging (B.8) and (B.9) back into (B.4) and using (A.3), we obtain (cid:18) ∇
Revised: Now plugging (B.8) and (B.9) back into (B.4) and using (A.3), we obtain (cid:18) ∇

Original: [Equation]
Revised: [Equation]

Original: L
Revised: L

Original: [Equation]
Revised: [Equation]

Original: η
Revised: η

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: In order to make the upper bound negative, we let
Revised: In order to make the upper bound negative, we let

Original: [Equation]
Revised: [Equation]

Original: and obtain
Revised: and obtain

Original: [Equation]
Revised: [Equation]

Original: This ends proof of Lemma 10.
Revised: This ends proof of Lemma 10.

Original: B.9. Proof of Lemma
Revised: B.9. Proof of Lemma

Original: Proof We suppose the event ∩ ∞ k =

Revised: Proof We suppose the event ∩ ∞ k =

Original: ( A k ∩ B k ) happens.
Revised: ( A k ∩ B k ) happens.

Original: We start from finding the lower bound of δ trial k η 1 ,k η 2 ,k .
Revised: We start from finding the lower bound of δ trial k η 1 ,k η 2 ,k .

Original: Since the updating rule of the adaptive parameters (2.9) increases η 1 ,k by a factor of ν 2 and decreases η 2 ,k by a factor of 1 /ν , we have that η 1 , 0 ≤ η 1 ,k and η 2 , 0 ≥ η 2 ,k for all k ≥ 0 .
Revised: Since the updating rule of the adaptive parameters (2.9) increases η 1 ,k by a factor of ν 2 and decreases η 2 ,k by a factor of 1 /ν , we have that η 1 , 0 ≤ η 1 ,k and η 2 , 0 ≥ η 2 ,k for all k ≥ 0 .

Original: Using this fact, we have that for any k ≥ 0 ,
Revised: Using this fact, we have that for any k ≥ 0 ,

Original: [Equation]
Revised: [Equation]

Original: Using the above display
Revised: Using the above display

Original: [Equation]
Revised: [Equation]

Original: we get
Revised: we get

Original: [Equation]
Revised: [Equation]

Original: and obtain
Revised: and obtain

Original: [Equation]
Revised: [Equation]

Original: Using Lemma 10 and (B.11)
Revised: Using Lemma 10 and (B.11)

Original: [Equation]
Revised: [Equation]

Original: we obtain the conditions for all adaptive parameters to be stabilized as
Revised: we obtain the conditions for all adaptive parameters to be stabilized as

Original: [Equation]
Revised: [Equation]

Original: Note that the lower bound of η 1 ,k η 2 ,k and the upper bound of η 2 ,k , δ k η 1 ,k /η 2 ,k do not depend on k .
Revised: Note that the lower bound of η 1 ,k η 2 ,k and the upper bound of η 2 ,k , δ k η 1 ,k /η 2 ,k do not depend on k .

Original: The updating rule of the adaptive parameters (2.9) implies that η 1 ,k η 2 ,k increases by a factor of ν , η 2 ,k decreases by a factor of 1 /ν , and
Revised: The updating rule of the adaptive parameters (2.9) implies that η 1 ,k η 2 ,k increases by a factor of ν , η 2 ,k decreases by a factor of 1 /ν , and

Original: δ k η 1 ,k /η 2 ,k decreases at least by a factor of 1 /ν .
Revised: δ k η 1 ,k /η 2 ,k decreases at least by a factor of 1 /ν .

Original: Thus, conditioned on the event ∩ ∞ k =

Revised: Thus, conditioned on the event ∩ ∞ k =

Original: ( A k ∩ B k ) , all parameters are stabilized after sufficiently large outer iterations k .
Revised: ( A k ∩ B k ) , all parameters are stabilized after sufficiently large outer iterations k .

Original: Now, using the fact that P ( A k | x k , λ k ) = 1 and P ( B k |A k , x k , λ k ) = 1 , we have P ( A k ∩ B k | x k , λ k ) = 1 .
Revised: Now, using the fact that P ( A k | x k , λ k ) = 1 and P ( B k |A k , x k , λ k ) = 1 , we have P ( A k ∩ B k | x k , λ k ) = 1 .

Original: Using Boole’s inequality,
Revised: Using Boole’s inequality,

Original: [Equation]
Revised: [Equation]

Original: Therefore, the event ∩ ∞ k =

Revised: Therefore, the event ∩ ∞ k =

Original: ( A k ∩B k ) happens with probability 1, hence, after sufficiently large outer iterations k , all parameters are stabilized almost surely.
Revised: ( A k ∩B k ) happens with probability 1, hence, after sufficiently large outer iterations k , all parameters are stabilized almost surely.

Original: This ends proof of Lemma 11.
Revised: This ends proof of Lemma 11.

Original: B.10. Proof of Lemma
Revised: B.10. Proof of Lemma

Original: Proof Let k ≥ 

Revised: Proof Let k ≥

Original: and we suppose the algorithm reaches ( x k , λ k ) and the event
Revised: and we suppose the algorithm reaches ( x k , λ k ) and the event

Original: A k ∩ B k happens.
Revised: A k ∩ B k happens.

Original: We start from establishing Lipschitz continuity of ∇L η .
Revised: We start from establishing Lipschitz continuity of ∇L η .

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: Using Assumption 1, we have H, G, ∇ x L , and c are all Lipschitz continuous and bounded over X .
Revised: Using Assumption 1, we have H, G, ∇ x L , and c are all Lipschitz continuous and bounded over X .

Original: Using this fact we have ∇L η is also Lipschitz continuous over X .
Revised: Using this fact we have ∇L η is also Lipschitz continuous over X .

Original: We denote Γ be the Lipschitz constant for ∇L η .
Revised: We denote Γ be the Lipschitz constant for ∇L η .

Original: Now we let C be a line segment given by the vector function s ( t ) =
Revised: Now we let C be a line segment given by the vector function s ( t ) =

Original: ( x k + tα k ˜∆ x k , λ k + tα k ˜∆ λ k ) where 0 ≤ t ≤ 1 .
Revised: ( x k + tα k ˜∆ x k , λ k + tα k ˜∆ λ k ) where 0 ≤ t ≤ 1 .

Original: Using this expression together with the fundamental theorem for line integrals, we get
Revised: Using this expression together with the fundamental theorem for line integrals, we get

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: k ˜∆ λ k (cid:19) + α k (cid:90) (cid:104) ∇L η
Revised: k ˜∆ λ k (cid:19) + α k (cid:90) (cid:104) ∇L η

Original: ( x k + tα k ˜∆ x k , λ k + tα k ˜∆ λ k ) − ∇L k η (cid:105) T (cid:18)
Revised: ( x k + tα k ˜∆ x k , λ k + tα k ˜∆ λ k ) − ∇L k η (cid:105) T (cid:18)

Original: ˜∆ x k ˜∆ λ k (cid:19) dt ≤ L k η + α k (cid:18) ∇ x L k η ∇ λ L k η (cid:19) T (cid:18)
Revised: ˜∆ x k ˜∆ λ k (cid:19) dt ≤ L k η + α k (cid:18) ∇ x L k η ∇ λ L k η (cid:19) T (cid:18)

Original: ˜∆ x k ˜∆ λ k (cid:19) + α k (cid:90)0 Γ tα k (cid:13)(cid:13)(cid:13) (cid:13) (cid:18)
Revised: ˜∆ x k ˜∆ λ k (cid:19) + α k (cid:90)0 Γ tα k (cid:13)(cid:13)(cid:13) (cid:13) (cid:18)

Original: ˜∆ x k ˜∆ λ k (cid:19)(cid:13)(cid:13)(cid:13) (cid:13) dt ≤ L k η + α k (cid:18) ∇ x L k η ∇ λ L k η (cid:19) T (cid:18)
Revised: ˜∆ x k ˜∆ λ k (cid:19)(cid:13)(cid:13)(cid:13) (cid:13) dt ≤ L k η + α k (cid:18) ∇ x L k η ∇ λ L k η (cid:19) T (cid:18)

Original: ˜∆ x k ˜∆ λ k (cid:19) + α 2 k Γ 2 (cid:13)(cid:13)(cid:13)(cid:13)(cid:18)
Revised: ˜∆ x k ˜∆ λ k (cid:19) + α 2 k Γ 2 (cid:13)(cid:13)(cid:13)(cid:13)(cid:18)

Original: ˜∆ x k ˜∆ λ k (cid:19)(cid:13)(cid:13)(cid:13)(cid:13) .
Revised: ˜∆ x k ˜∆ λ k (cid:19)(cid:13)(cid:13)(cid:13)(cid:13) .

Original: A
Revised: A

Original: Using (A.3), we have
Revised: Using (A.3), we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: B
Revised: B

Original: [Equation]
Revised: [Equation]

Original: Using this expression together with Lemma
Revised: Using this expression together with Lemma

Original: [Equation]
Revised: [Equation]

Original: we get
Revised: we get

Original: [Equation]
Revised: [Equation]

Original: where η ⋆ 2 is the stabilized value of η 2 .
Revised: where η ⋆ 2 is the stabilized value of η 2 .

Original: Using the above display, if
Revised: Using the above display, if

Original: [Equation]
Revised: [Equation]

Original: then the Armijo condition is satisfied.
Revised: then the Armijo condition is satisfied.

Original: Moreover, since the upper bound of α k does not depend on k , we can find l ≥ 0 , independent of k , such that for any k , 0 < ( ρ ) l ≤ (1 − β ) η 2 4ΓΨ.
Revised: Moreover, since the upper bound of α k does not depend on k , we can find l ≥ 0 , independent of k , such that for any k , 0 < ( ρ ) l ≤ (1 − β ) η 2 4ΓΨ.

Original: Finally, if we let α min = ( ρ ) l , then for any outer iteration k , we have 0 < α min ≤ α k .
Revised: Finally, if we let α min = ( ρ ) l , then for any outer iteration k , we have 0 < α min ≤ α k .

Original: This ends proof of Lemma 12.
Revised: This ends proof of Lemma 12.

Original: B.11. Proof of Lemma
Revised: B.11. Proof of Lemma

Original: Proof We suppose the event ∩ ∞ k =

Revised: Proof We suppose the event ∩ ∞ k =

Original: ( A k ∩ B k ) happens.
Revised: ( A k ∩ B k ) happens.

Original: Using Assumption
Revised: Using Assumption

Original: [Equation]
Revised: [Equation]

Original: we let k f
Revised: we let k f

Original: [Equation]
Revised: [Equation]

Original: k c , k g > 0 be constants such that | f k | ≤ k f , ∥ c k ∥ ≤ k c
Revised: k c , k g > 0 be constants such that | f k | ≤ k f , ∥ c k ∥ ≤ k c

Original: [Equation]
Revised: [Equation]

Original: and ∥∇ f k ∥ ≤ k g .
Revised: and ∥∇ f k ∥ ≤ k g .

Original: Using this fact together with Lemma 5 and
Revised: Using this fact together with Lemma 5 and

Original: Cauchy-Schwarz inequality
Revised: Cauchy-Schwarz inequality

Original: [Equation]
Revised: [Equation]

Original: we get for any k ≥ 0 ,
Revised: we get for any k ≥ 0 ,

Original: [Equation]
Revised: [Equation]

Original: where η ⋆ 2 is the stabilized value of η 2 .
Revised: where η ⋆ 2 is the stabilized value of η 2 .

Original: Using Lemma 10 and 12, we have that L η ( x k , λ k ) ≤ L η ( x 0 , λ 0 ) for all k ≥ 0 .
Revised: Using Lemma 10 and 12, we have that L η ( x k , λ k ) ≤ L η ( x 0 , λ 0 ) for all k ≥ 0 .

Original: This leads to
Revised: This leads to

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: If we let K 1 =
Revised: If we let K 1 =

Original: [Equation]
Revised: [Equation]

Original: > 0 , K
Revised: > 0 , K

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: k
Revised: k

Original: [Equation]
Revised: [Equation]

Original: + η 2 ,
Revised: + η 2 ,

Original: [Equation]
Revised: [Equation]

Original: Υ
Revised: Υ

Original: [Equation]
Revised: [Equation]

Original: > 0 , and K 3 =
Revised: > 0 , and K 3 =

Original: |L
Revised: |L

Original: [Equation]
Revised: [Equation]

Original: ( x 0 , λ 0 ) | + k f > 0 , then we get for any k ≥ 0 ,
Revised: ( x 0 , λ 0 ) | + k f > 0 , then we get for any k ≥ 0 ,

Original: [Equation]
Revised: [Equation]

Original: This implies that { λ k } k ≥ 0 is bounded.
Revised: This implies that { λ k } k ≥ 0 is bounded.

Original: Using (B.13), the event ∩ ∞ k =

Revised: Using (B.13), the event ∩ ∞ k =

Original: ( A k ∩ B k ) happens with probability 1, hence, { λ k } k ≥ 0 is bounded almost surely.
Revised: ( A k ∩ B k ) happens with probability 1, hence, { λ k } k ≥ 0 is bounded almost surely.

Original: This ends proof of Lemma 13.
Revised: This ends proof of Lemma 13.

Original: B.12. Proof of Theorem
Revised: B.12. Proof of Theorem

Original: Proof We suppose the event ∩ ∞ k =

Revised: Proof We suppose the event ∩ ∞ k =

Original: ( A k ∩ B k ) happens.
Revised: ( A k ∩ B k ) happens.

Original: Using Lemma 10 and 12, we have that for any k ≥ 0 ,
Revised: Using Lemma 10 and 12, we have that for any k ≥ 0 ,

Original: [Equation]
Revised: [Equation]

Original: ∇ x L k η
Revised: ∇ x L k η

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: for some α k ∈ (0 , 1] .
Revised: for some α k ∈ (0 , 1] .

Original: Combining the above displays, we have that for any k ≥ 0 ,
Revised: Combining the above displays, we have that for any k ≥ 0 ,

Original: [Equation]
Revised: [Equation]

Original: where η ⋆ 2 is the stabilized value of η 2 .
Revised: where η ⋆ 2 is the stabilized value of η 2 .

Original: Summing over k , we have ∞ (cid:88)
Revised: Summing over k , we have ∞ (cid:88)

Original: [Equation]
Revised: [Equation]

Original: Therefore, ∥∇L k ∥ → 0 as k → ∞ .
Revised: Therefore, ∥∇L k ∥ → 0 as k → ∞ .

Original: Using (B.13), the event ∩ ∞ k =

Revised: Using (B.13), the event ∩ ∞ k =

Original: ( A k ∩ B k ) happens with probability 1, hence, P ( ∥∇L k ∥ → 0 as k → ∞ )
Revised: ( A k ∩ B k ) happens with probability 1, hence, P ( ∥∇L k ∥ → 0 as k → ∞ )

Original: = 1 .
Revised: = 1 .

Original: This ends proof of Theorem 1.
Revised: This ends proof of Theorem 1.

Original: B.13. Proof of Theorem
Revised: B.13. Proof of Theorem

Original: Proof We suppose the event ∩ ∞ k =

Revised: Proof We suppose the event ∩ ∞ k =

Original: ( A k ∩B k ) happens.
Revised: ( A k ∩B k ) happens.

Original: We first show for all sufficiently large k , almost surely, unit stepsize is admissible.
Revised: We first show for all sufficiently large k , almost surely, unit stepsize is admissible.

Original: It suffices to show that for all sufficiently large k ,
Revised: It suffices to show that for all sufficiently large k ,

Original: [Equation]
Revised: [Equation]

Original: almost surely .
Revised: almost surely .

Original: [Equation]
Revised: [Equation]

Original: Using the fact that for any k ≥
Revised: Using the fact that for any k ≥

Original: [Equation]
Revised: [Equation]

Original: where M ∈ R m × n and c ∈ R n
Revised: where M ∈ R m × n and c ∈ R n

Original: [Equation]
Revised: [Equation]

Original: we get
Revised: we get

Original: [Equation]
Revised: [Equation]

Original: where ∇ x H k = ∇ 3 xxx f k + (cid:80) mi =1 λ i,k ∇ 3 xxx c
Revised: where ∇ x H k = ∇ 3 xxx f k + (cid:80) mi =1 λ i,k ∇ 3 xxx c

Original: i,k and ∇ λ H k = ∇ 2 xx c k .
Revised: i,k and ∇ λ H k = ∇ 2 xx c k .

Original: Using Assumption 4, we have that the third derivatives of f and c are continuous, hence, ∇ 2 L k η is continuous over X .
Revised: Using Assumption 4, we have that the third derivatives of f and c are continuous, hence, ∇ 2 L k η is continuous over X .

Original: Now we let
Revised: Now we let

Original: H k = (cid:18)
Revised: H k = (cid:18)

Original: H k + η 2 ,k H 2 k + η 1 ,
Revised: H k + η 2 ,k H 2 k + η 1 ,

Original: k G Tk G k
Revised: k G Tk G k

Original: G Tk + η 2 ,
Revised: G Tk + η 2 ,

Original: [Equation]
Revised: [Equation]

Original: H
Revised: H

Original: [Equation]
Revised: [Equation]

Original: G Tk
Revised: G Tk

Original: G k + η 2 ,k G k H k η 2 ,
Revised: G k + η 2 ,k G k H k η 2 ,

Original: [Equation]
Revised: [Equation]

Original: G k G Tk (cid:19) .
Revised: G k G Tk (cid:19) .

Original: Using ∥∇L k ∥ = (cid:13)(cid:13)(cid:13)(cid:13)(cid:18) ∇ x L k c k
Revised: Using ∥∇L k ∥ = (cid:13)(cid:13)(cid:13)(cid:13)(cid:18) ∇ x L k c k

Original: (cid:19)(cid:13)(cid:13)(cid:13)(cid:13) =
Revised: (cid:19)(cid:13)(cid:13)(cid:13)(cid:13) =

Original: o (1) , we have ∇ 2 L k η = H k
Revised: o (1) , we have ∇ 2 L k η = H k

Original: + o (1) .
Revised: + o (1) .

Original: Applying Taylor’s theorem to the augmented Lagrangian merit function about ( x k , λ k ) yields
Revised: Applying Taylor’s theorem to the augmented Lagrangian merit function about ( x k , λ k ) yields

Original: L
Revised: L

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ˜∆ x
Revised: ˜∆ x

Original: [Equation]
Revised: [Equation]

Original: λ k + ˜∆ λ k ) ≤ L
Revised: λ k + ˜∆ λ k ) ≤ L

Original: [Equation]
Revised: [Equation]

Original: + (cid:18) ∇ x L k η ∇
Revised: + (cid:18) ∇ x L k η ∇

Original: [Equation]
Revised: [Equation]

Original: L
Revised: L

Original: [Equation]
Revised: [Equation]

Original: (cid:19)
Revised: (cid:19)

Original: [Equation]
Revised: [Equation]

Original: (cid:18)
Revised: (cid:18)

Original: ˜∆
Revised: ˜∆

Original: [Equation]
Revised: [Equation]

Original: k ˜∆ λ
Revised: k ˜∆ λ

Original: [Equation]
Revised: [Equation]

Original: (cid:19)
Revised: (cid:19)

Original: + (cid:18)
Revised: + (cid:18)

Original: ˜∆ x k ˜∆ λ k (cid:19) T ∇
Revised: ˜∆ x k ˜∆ λ k (cid:19) T ∇

Original: [Equation]
Revised: [Equation]

Original: L
Revised: L

Original: [Equation]
Revised: [Equation]

Original: (cid:18) ˜∆ x k ˜∆ λ k (cid:19) + o (cid:32)(cid:13)(cid:13)(cid:13)(cid:13)(cid:18)
Revised: (cid:18) ˜∆ x k ˜∆ λ k (cid:19) + o (cid:32)(cid:13)(cid:13)(cid:13)(cid:13)(cid:18)

Original: ˜∆ x k ˜∆ λ k (cid:19)(cid:13)(cid:13)(cid:13)(cid:13) 2 (cid:33) =
Revised: ˜∆ x k ˜∆ λ k (cid:19)(cid:13)(cid:13)(cid:13)(cid:13) 2 (cid:33) =

Original: L k η + (cid:18) ∇ x L k η ∇ λ L k η (cid:19) T (cid:18)
Revised: L k η + (cid:18) ∇ x L k η ∇ λ L k η (cid:19) T (cid:18)

Original: ˜∆ x k ˜∆ λ k (cid:19) + (cid:18)
Revised: ˜∆ x k ˜∆ λ k (cid:19) + (cid:18)

Original: ˜∆ x k ˜∆ λ k (cid:19) T
Revised: ˜∆ x k ˜∆ λ k (cid:19) T

Original: H k (cid:18)
Revised: H k (cid:18)

Original: ˜∆ x k ˜∆ λ k (cid:19) + o (cid:32)(cid:13)(cid:13)(cid:13) (cid:13) (cid:18)
Revised: ˜∆ x k ˜∆ λ k (cid:19) + o (cid:32)(cid:13)(cid:13)(cid:13) (cid:13) (cid:18)

Original: ˜∆ x k ˜∆ λ k (cid:19)(cid:13)(cid:13)(cid:13) (cid:13) 2 (cid:33) =
Revised: ˜∆ x k ˜∆ λ k (cid:19)(cid:13)(cid:13)(cid:13) (cid:13) 2 (cid:33) =

Original: L k η + 1(cid:18) ∇ x L k η ∇ λ L k η (cid:19) T (cid:18)
Revised: L k η + 1(cid:18) ∇ x L k η ∇ λ L k η (cid:19) T (cid:18)

Original: ˜∆ x k ˜∆ λ k (cid:19) + (cid:18)
Revised: ˜∆ x k ˜∆ λ k (cid:19) + (cid:18)

Original: ˜∆ x k ˜∆ λ k (cid:19) T
Revised: ˜∆ x k ˜∆ λ k (cid:19) T

Original: H k (cid:18)
Revised: H k (cid:18)

Original: ˜∆ x k ˜∆ λ k (cid:19) + (cid:18) ∇ x L k η ∇ λ L k η (cid:19) T (cid:18)
Revised: ˜∆ x k ˜∆ λ k (cid:19) + (cid:18) ∇ x L k η ∇ λ L k η (cid:19) T (cid:18)

Original: ˜∆ x k ˜∆ λ k (cid:19) + o (cid:32)(cid:13)(cid:13)(cid:13)(cid:13)(cid:18)
Revised: ˜∆ x k ˜∆ λ k (cid:19) + o (cid:32)(cid:13)(cid:13)(cid:13)(cid:13)(cid:18)

Original: ˜∆ x k ˜∆ λ k (cid:19)(cid:13)(cid:13)(cid:13)(cid:13) 2 (cid:33) =
Revised: ˜∆ x k ˜∆ λ k (cid:19)(cid:13)(cid:13)(cid:13)(cid:13) 2 (cid:33) =

Original: L k η + 1(cid:18) ∇ x L k η ∇ λ L k η (cid:19) T (cid:18)
Revised: L k η + 1(cid:18) ∇ x L k η ∇ λ L k η (cid:19) T (cid:18)

Original: ˜∆ x k ˜∆ λ k (cid:19) + (cid:18)
Revised: ˜∆ x k ˜∆ λ k (cid:19) + (cid:18)

Original: ˜∆ x k ˜∆ λ k (cid:19) T
Revised: ˜∆ x k ˜∆ λ k (cid:19) T

Original: H k (cid:18)
Revised: H k (cid:18)

Original: ˜∆ x k ˜∆ λ k (cid:19) − (cid:18)
Revised: ˜∆ x k ˜∆ λ k (cid:19) − (cid:18)

Original: ˜∆ x k ˜∆ λ k (cid:19) T (cid:18)
Revised: ˜∆ x k ˜∆ λ k (cid:19) T (cid:18)

Original: ( I + η 2 ,k H k ) B k + η 1 ,k G
Revised: ( I + η 2 ,k H k ) B k + η 1 ,k G

Original: Tk G k
Revised: Tk G k

Original: ( I + η 2 ,k H k ) G Tk
Revised: ( I + η 2 ,k H k ) G Tk

Original: G k
Revised: G k

Original: ( I + η 2 ,k B k ) η 2 ,k G k G Tk (cid:19) (cid:18)
Revised: ( I + η 2 ,k B k ) η 2 ,k G k G Tk (cid:19) (cid:18)

Original: ∆ x k ∆ λ k (cid:19) + o (cid:32)(cid:13)(cid:13)(cid:13)(cid:13)(cid:18)
Revised: ∆ x k ∆ λ k (cid:19) + o (cid:32)(cid:13)(cid:13)(cid:13)(cid:13)(cid:18)

Original: ˜∆ x k ˜∆ λ k (cid:19)(cid:13)(cid:13)(cid:13)(cid:13) 2 (cid:33) =
Revised: ˜∆ x k ˜∆ λ k (cid:19)(cid:13)(cid:13)(cid:13)(cid:13) 2 (cid:33) =

Original: L k η + 1(cid:18) ∇ x L k η ∇ λ L k η (cid:19) T (cid:18)
Revised: L k η + 1(cid:18) ∇ x L k η ∇ λ L k η (cid:19) T (cid:18)

Original: ˜∆ x k ˜∆ λ k (cid:19) + (cid:18)
Revised: ˜∆ x k ˜∆ λ k (cid:19) + (cid:18)

Original: ˜∆ x k ˜∆ λ k (cid:19) T
Revised: ˜∆ x k ˜∆ λ k (cid:19) T

Original: H k (cid:18)
Revised: H k (cid:18)

Original: ˜∆ x k − ∆ x k ˜∆ λ k − ∆ λ k (cid:19) + (cid:18) ˜∆ x k ˜∆ λ k (cid:19) T (cid:18)
Revised: ˜∆ x k − ∆ x k ˜∆ λ k − ∆ λ k (cid:19) + (cid:18) ˜∆ x k ˜∆ λ k (cid:19) T (cid:18)

Original: H k − (cid:18)
Revised: H k − (cid:18)

Original: ( I + η 2 ,k H k ) B k + η 1 ,k G
Revised: ( I + η 2 ,k H k ) B k + η 1 ,k G

Original: Tk G k
Revised: Tk G k

Original: ( I + η 2 ,k H k ) G Tk
Revised: ( I + η 2 ,k H k ) G Tk

Original: G k
Revised: G k

Original: ( I + η 2 ,k B k ) η 2 ,k G k G Tk (cid:19)(cid:19) (cid:18)
Revised: ( I + η 2 ,k B k ) η 2 ,k G k G Tk (cid:19)(cid:19) (cid:18)

Original: ∆ x k ∆ λ k (cid:19) + o (cid:32)(cid:13)(cid:13)(cid:13)(cid:13)(cid:18)
Revised: ∆ x k ∆ λ k (cid:19) + o (cid:32)(cid:13)(cid:13)(cid:13)(cid:13)(cid:18)

Original: ˜∆ x k ˜∆ λ k (cid:19)(cid:13)(cid:13)(cid:13)(cid:13) 2 (cid:33) .
Revised: ˜∆ x k ˜∆ λ k (cid:19)(cid:13)(cid:13)(cid:13)(cid:13) 2 (cid:33) .

Original: This leads to
Revised: This leads to

Original: [Equation]
Revised: [Equation]

Original: Now we let Υ k =
Revised: Now we let Υ k =

Original: ∥ B k ∥∨∥ H k
Revised: ∥ B k ∥∨∥ H k

Original: ∥∨∥ G k ∥ .
Revised: ∥∨∥ G k ∥ .

Original: Using Assumption
Revised: Using Assumption

Original: [Equation]
Revised: [Equation]

Original: we get ∥ ( H k − B k )∆ x k ∥ ≤ ∥ ( H k − B k ) ∥
Revised: we get ∥ ( H k − B k )∆ x k ∥ ≤ ∥ ( H k − B k ) ∥

Original: ∥ ∆ x k ∥
Revised: ∥ ∆ x k ∥

Original: =
Revised: =

Original: o ( ∥ ∆ x k ∥ ) .
Revised: o ( ∥ ∆ x k ∥ ) .

Original: Using this expression together with (A.3), (B.14), and o ( ∥ ( ˜∆ x k , ˜∆ λ k ) ∥ )
Revised: Using this expression together with (A.3), (B.14), and o ( ∥ ( ˜∆ x k , ˜∆ λ k ) ∥ )

Original: =
Revised: =

Original: o ( ∥ (∆ x k , ∆ λ k ) ∥ ) , we have that for any k ≥ 0 ,
Revised: o ( ∥ (∆ x k , ∆ λ k ) ∥ ) , we have that for any k ≥ 0 ,

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: + (1 + 2 η 2 ,k Υ k ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:18) ∆ x
Revised: + (1 + 2 η 2 ,k Υ k ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:18) ∆ x

Original: [Equation]
Revised: [Equation]

Original: (cid:19)(cid:13)(cid:13)(cid:13)(cid:13) o
Revised: (cid:19)(cid:13)(cid:13)(cid:13)(cid:13) o

Original: [Equation]
Revised: [Equation]

Original: + o (cid:32)(cid:13)(cid:13)(cid:13)(cid:13)(cid:18)
Revised: + o (cid:32)(cid:13)(cid:13)(cid:13)(cid:13)(cid:18)

Original: ˜∆ x k ˜∆ λ k (cid:19)(cid:13)(cid:13)(cid:13)(cid:13) 2 (cid:33) ≤ L k
Revised: ˜∆ x k ˜∆ λ k (cid:19)(cid:13)(cid:13)(cid:13)(cid:13) 2 (cid:33) ≤ L k

Original: [Equation]
Revised: [Equation]

Original: + 1(cid:18)
Revised: + 1(cid:18)

Original: [Equation]
Revised: [Equation]

Original: η
Revised: η

Original: [Equation]
Revised: [Equation]

Original: (cid:19)
Revised: (cid:19)

Original: [Equation]
Revised: [Equation]

Original: (cid:19)
Revised: (cid:19)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (cid:13)(cid:13)(cid:13)(cid:13)(cid:18)
Revised: (cid:13)(cid:13)(cid:13)(cid:13)(cid:18)

Original: [Equation]
Revised: [Equation]

Original: (cid:19)(cid:13)(cid:13)(cid:13)(cid:13) + o (cid:32)(cid:13)(cid:13)(cid:13)(cid:13)(cid:18) ∆ x k
Revised: (cid:19)(cid:13)(cid:13)(cid:13)(cid:13) + o (cid:32)(cid:13)(cid:13)(cid:13)(cid:13)(cid:18) ∆ x k

Original: [Equation]
Revised: [Equation]

Original: λ k (cid:19)(cid:13)(cid:13)(cid:13)(cid:13) 2 (cid:33) .
Revised: λ k (cid:19)(cid:13)(cid:13)(cid:13)(cid:13) 2 (cid:33) .

Original: (B.17)
Revised: (B.17)

Original: Using the fact that for any k ≥ 0 , δ k ≤ δ trial k = (cid:18)2 − β (cid:19) η 2 ,k 2Ψ 2 k (3Υ k + 4 η 2 ,k Υ 2 k + η 1 ,k Υ 2 k ) and Lemma 10, we have that for any k ≥ 0 , δ k
Revised: Using the fact that for any k ≥ 0 , δ k ≤ δ trial k = (cid:18)2 − β (cid:19) η 2 ,k 2Ψ 2 k (3Υ k + 4 η 2 ,k Υ 2 k + η 1 ,k Υ 2 k ) and Lemma 10, we have that for any k ≥ 0 , δ k

Original: (3Υ k + 4 η 2 ,k Υ 2 k
Revised: (3Υ k + 4 η 2 ,k Υ 2 k

Original: + η 1 ,k Υ 2 k ) ≤ (cid:18)2 − β (cid:19)
Revised: + η 1 ,k Υ 2 k ) ≤ (cid:18)2 − β (cid:19)

Original: η 2 ,k 2Ψ 2 k ⇒ δ k
Revised: η 2 ,k 2Ψ 2 k ⇒ δ k

Original: (3Υ k + 4 η 2 ,k
Revised: (3Υ k + 4 η 2 ,k

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: (cid:19)(cid:13)(cid:13)(cid:13)(cid:13)
Revised: (cid:19)(cid:13)(cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: (cid:19)
Revised: (cid:19)

Original: [Equation]
Revised: [Equation]

Original: (cid:18)
Revised: (cid:18)

Original: [Equation]
Revised: [Equation]

Original: ˜∆ λ k (cid:19) .
Revised: ˜∆ λ k (cid:19) .

Original: We let K 1 ≥ 0 be the outer iteration such that for any k ≥ K 1 ,
Revised: We let K 1 ≥ 0 be the outer iteration such that for any k ≥ K 1 ,

Original: [Equation]
Revised: [Equation]

Original: Plugging the above inequality back into (B.17), we have that for any k ≥ K 1 ,
Revised: Plugging the above inequality back into (B.17), we have that for any k ≥ K 1 ,

Original: [Equation]
Revised: [Equation]

Original: Using (B.13), the event ∩ ∞ k =

Revised: Using (B.13), the event ∩ ∞ k =

Original: ( A k ∩ B k ) happens with probability 1, hence,
Revised: ( A k ∩ B k ) happens with probability 1, hence,

Original: [Equation]
Revised: [Equation]

Original: Next, we show for all sufficiently large k ,
Revised: Next, we show for all sufficiently large k ,

Original: [Equation]
Revised: [Equation]

Original: almost surely , where δ ⋆ be the stabilized value of δ ∈ (0 , 1) .
Revised: almost surely , where δ ⋆ be the stabilized value of δ ∈ (0 , 1) .

Original: We start from dividing (cid:18)
Revised: We start from dividing (cid:18)

Original: x
Revised: x

Original: k + ˜∆ x k − x ⋆ λ k + ˜∆ λ k − λ
Revised: k + ˜∆ x k − x ⋆ λ k + ˜∆ λ k − λ

Original: ⋆ (cid:19) into two terms as follows:
Revised: ⋆ (cid:19) into two terms as follows:

Original: [Equation]
Revised: [Equation]

Original: First we develop the first term in (B.18).
Revised: First we develop the first term in (B.18).

Original: Using Assumption 1– 2 together with ∇L ⋆ = 0 , we obtain for any k ≥ 0 , (cid:18)
Revised: Using Assumption 1– 2 together with ∇L ⋆ = 0 , we obtain for any k ≥ 0 , (cid:18)

Original: [Equation]
Revised: [Equation]

Original: Using Assumption 1, we know ∇ 2 L is continuous over X .
Revised: Using Assumption 1, we know ∇ 2 L is continuous over X .

Original: Using this fact, we apply Taylor’s theorem and obtain ∇L
Revised: Using this fact, we apply Taylor’s theorem and obtain ∇L

Original: [Equation]
Revised: [Equation]

Original: (cid:18) H ( x k + t ( x ⋆ − x k ) , λ k + t
Revised: (cid:18) H ( x k + t ( x ⋆ − x k ) , λ k + t

Original: ( λ ⋆ − λ k ))
Revised: ( λ ⋆ − λ k ))

Original: G T ( x k + t
Revised: G T ( x k + t

Original: x ⋆ − x k ))
Revised: x ⋆ − x k ))

Original: G ( x k + t ( x ⋆ − x k ))
Revised: G ( x k + t ( x ⋆ − x k ))

Original: (cid:19) (cid:18) x
Revised: (cid:19) (cid:18) x

Original: k − x ⋆ λ k − λ ⋆ (cid:19) dt.
Revised: k − x ⋆ λ k − λ ⋆ (cid:19) dt.

Original: Now we let H ( t ) =
Revised: Now we let H ( t ) =

Original: H ( x k + t
Revised: H ( x k + t

Original: ( x ⋆ − x k ) , λ k + t ( λ ⋆ − λ k )) and G ( t )
Revised: ( x ⋆ − x k ) , λ k + t ( λ ⋆ − λ k )) and G ( t )

Original: =
Revised: =

Original: G ( x k + t
Revised: G ( x k + t

Original: ( x ⋆ − x k )) .
Revised: ( x ⋆ − x k )) .

Original: Then we rewrite the above display as ∇L k − ∇L ⋆ = (cid:90) (cid:18)
Revised: Then we rewrite the above display as ∇L k − ∇L ⋆ = (cid:90) (cid:18)

Original: [Equation]
Revised: [Equation]

Original: ( t ) G T ( t )
Revised: ( t ) G T ( t )

Original: G ( t ) (cid:19) (cid:18) x
Revised: G ( t ) (cid:19) (cid:18) x

Original: [Equation]
Revised: [Equation]

Original: x ⋆ λ k
Revised: x ⋆ λ k

Original: [Equation]
Revised: [Equation]

Original: λ ⋆ (cid:19) dt.
Revised: λ ⋆ (cid:19) dt.

Original: Plugging the above display back into (B.19), we obtain for any k ≥ 0 , (cid:18) x k + ∆ x k − x ⋆ λ k + ∆ λ k − λ ⋆ (cid:19) =
Revised: Plugging the above display back into (B.19), we obtain for any k ≥ 0 , (cid:18) x k + ∆ x k − x ⋆ λ k + ∆ λ k − λ ⋆ (cid:19) =

Original: (cid:18) B k
Revised: (cid:18) B k

Original: G Tk
Revised: G Tk

Original: G k (cid:19) − 1 (cid:18)(cid:18)
Revised: G k (cid:19) − 1 (cid:18)(cid:18)

Original: B k
Revised: B k

Original: G Tk
Revised: G Tk

Original: G k (cid:19) (cid:18) x k − x ⋆ λ k − λ ⋆ (cid:19) − ( ∇L k − ∇L ⋆ ) (cid:19) =
Revised: G k (cid:19) (cid:18) x k − x ⋆ λ k − λ ⋆ (cid:19) − ( ∇L k − ∇L ⋆ ) (cid:19) =

Original: (cid:18) B k
Revised: (cid:18) B k

Original: G Tk
Revised: G Tk

Original: G k (cid:19) − 1 (cid:18)(cid:90) (cid:18) B k − H ( t ) G
Revised: G k (cid:19) − 1 (cid:18)(cid:90) (cid:18) B k − H ( t ) G

Original: Tk − G ( t ) T
Revised: Tk − G ( t ) T

Original: G k − G ( t ) (cid:19) (cid:18)
Revised: G k − G ( t ) (cid:19) (cid:18)

Original: x k − x ⋆ λ k − λ ⋆ (cid:19) dt (cid:19)
Revised: x k − x ⋆ λ k − λ ⋆ (cid:19) dt (cid:19)

Original: Using Assumption 1, we know H and G are Lipschitz continuous over X .
Revised: Using Assumption 1, we know H and G are Lipschitz continuous over X .

Original: Let Γ 1 , Γ 2 > 0 be the Lipschitz constants for H and G respectively.
Revised: Let Γ 1 , Γ 2 > 0 be the Lipschitz constants for H and G respectively.

Original: Using this fact together with Lemma 6 and Assumption 5, and taking ℓ
Revised: Using this fact together with Lemma 6 and Assumption 5, and taking ℓ

Original: [Table]
Revised: [Table]

Original: Using the above inequality together with (A.3) and (B.20), and taking ℓ 2 norm on both sides of (B.18), we have that for any k ≥ 0 ,
Revised: Using the above inequality together with (A.3) and (B.20), and taking ℓ 2 norm on both sides of (B.18), we have that for any k ≥ 0 ,

Original: [Equation]
Revised: [Equation]

Original: (cid:13) (cid:13)(cid:13)(cid:19) .
Revised: (cid:13) (cid:13)(cid:13)(cid:19) .

Original: We let ¯ K ≥ 0 be the outer iteration such that (cid:13)(cid:13)(cid:13)(cid:13)(cid:18) x k +
Revised: We let ¯ K ≥ 0 be the outer iteration such that (cid:13)(cid:13)(cid:13)(cid:13)(cid:18) x k +

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: x ⋆ λ
Revised: x ⋆ λ

Original: [Equation]
Revised: [Equation]

Original: (cid:19)(cid:13)(cid:13)(cid:13)(cid:13) holds where δ ⋆ be the stabilized value of δ ∈ (0 , 1) .
Revised: (cid:19)(cid:13)(cid:13)(cid:13)(cid:13) holds where δ ⋆ be the stabilized value of δ ∈ (0 , 1) .

Original: Now we let K 2 =
Revised: Now we let K 2 =

Original: ¯ K ∨ K 1 .
Revised: ¯ K ∨ K 1 .

Original: Then, we have for any k ≥ K 2 ,
Revised: Then, we have for any k ≥ K 2 ,

Original: [Equation]
Revised: [Equation]

Original: Using (B.13), the event ∩ ∞ k =

Revised: Using (B.13), the event ∩ ∞ k =

Original: ( A k ∩ B k ) happens with probability 1, hence,
Revised: ( A k ∩ B k ) happens with probability 1, hence,

Original: P (cid:18)
Revised: P (cid:18)

Original: [Equation]
Revised: [Equation]

Original: ∞ k = K(cid:26) (cid:13) (cid:13) (cid:13) (cid:13) (cid:18) x k
Revised: ∞ k = K(cid:26) (cid:13) (cid:13) (cid:13) (cid:13) (cid:18) x k

Original: [Equation]
Revised: [Equation]

Original: − x ⋆ λ k
Revised: − x ⋆ λ k

Original: [Equation]
Revised: [Equation]

Original: − λ
Revised: − λ

Original: [Equation]
Revised: [Equation]

Original: (cid:19)(cid:13)
Revised: (cid:19)(cid:13)

Original: [Table]
Revised: [Table]

Original: This ends proof of Theorem 2.
Revised: This ends proof of Theorem 2.

Original: Appendix C. Algorithms
Revised: Appendix C. Algorithms

Original: Algorithm
Revised: Algorithm

Original: Input: Initial iterate ( x 0 , λ 0 ) ;
Revised: Input: Initial iterate ( x 0 , λ 0 ) ;

Original: Scalars η 1 , 0 , η 2 , 0 , δ 0 ∈ (0 , 1) ; ξ B ∈ (0 , 1] , β ∈ (0 , 1 / 2) ; ρ ∈ (0 , 1) ; ν > 1 ; for k = 0 , 1 , 2 , . . .
Revised: Scalars η 1 , 0 , η 2 , 0 , δ 0 ∈ (0 , 1) ; ξ B ∈ (0 , 1] , β ∈ (0 , 1 / 2) ; ρ ∈ (0 , 1) ; ν > 1 ; for k = 0 , 1 , 2 , . . .

Original: do
Revised: do

Original: Compute f k , ∇ f k , c k , G k , H k , and generate B k
Revised: Compute f k , ∇ f k , c k , G k , H k , and generate B k

Original: Compute δ trial k by (2.6)
Revised: Compute δ trial k by (2.6)

Original: Set ( ˜∆ x k , ˜∆ λ k ) ← ( 0 , 0 ) and compute r k by (2.4)
Revised: Set ( ˜∆ x k , ˜∆ λ k ) ← ( 0 , 0 ) and compute r k by (2.4)

Original: Set δ k ← ( δ k
Revised: Set δ k ← ( δ k

Original: ∧
Revised: ∧

Original: δ trial k ) while Step Acceptance Condition does not hold do while ∥ r k ∥ >
Revised: δ trial k ) while Step Acceptance Condition does not hold do while ∥ r k ∥ >

Original: δ k ∥∇L k ∥
Revised: δ k ∥∇L k ∥

Original: ∥
Revised: ∥

Original: Γ k ∥
Revised: Γ k ∥

Original: Ψ k do
Revised: Ψ k do

Original: Generate S ∼ P , update ( ˜ ∆ x k , ˜ ∆ λ k ) by (2.5), and compute r k by (2.4) end if (cid:18) ∇ x L k η ∇ λ L k η (cid:19) T (cid:18)
Revised: Generate S ∼ P , update ( ˜ ∆ x k , ˜ ∆ λ k ) by (2.5), and compute r k by (2.4) end if (cid:18) ∇ x L k η ∇ λ L k η (cid:19) T (cid:18)

Original: ˜∆ x k ˜∆ λ k (cid:19) > − η 2 ,k 2 ∥∇L k ∥ 2 then
Revised: ˜∆ x k ˜∆ λ k (cid:19) > − η 2 ,k 2 ∥∇L k ∥ 2 then

Original: Set η 1 ,k ← η 1 ,k ν 2 , η 2 ,k ← η 2 ,k /ν
Revised: Set η 1 ,k ← η 1 ,k ν 2 , η 2 ,k ← η 2 ,k /ν

Original: Update δ trial k by (2.6) and set δ k ← ( δ k /ν 

Revised: Update δ trial k by (2.6) and set δ k ← ( δ k /ν

Original: ∧
Revised: ∧

Original: δ trial k ) end end
Revised: δ trial k ) end end

Original: Select α k to satisfy (2.10) using backtracking
Revised: Select α k to satisfy (2.10) using backtracking

Original: Update iterate by (2.11)
Revised: Update iterate by (2.11)

Original: Set η 1 ,k +1 ← η 1 ,k , η 2 ,k +1 ← η 2 ,k , and δ k +1 ← δ k end
Revised: Set η 1 ,k +1 ← η 1 ,k , η 2 ,k +1 ← η 2 ,k , and δ k +1 ← δ k end

Original: NEXACT
Revised: NEXACT

Original: Algorithm 2 and 3 use the ℓ 1 penalized merit function of the form ϕ π ( x ) =
Revised: Algorithm 2 and 3 use the ℓ 1 penalized merit function of the form ϕ π ( x ) =

Original: f ( x ) + π ∥ c ( x ) ∥ 1 .
Revised: f ( x ) + π ∥ c ( x ) ∥ 1 .

Original: Since ϕ π ( x ) is not differentiable and its directional derivative is hard to compute, we use the upper bound of the directional derivative of the merit function ϕ π along a step ˜∆ x k ,
Revised: Since ϕ π ( x ) is not differentiable and its directional derivative is hard to compute, we use the upper bound of the directional derivative of the merit function ϕ π along a step ˜∆ x k ,

Original: [Equation]
Revised: [Equation]

Original: ϕ ( ˜∆
Revised: ϕ ( ˜∆

Original: [Equation]
Revised: [Equation]

Original: 1 ) , when we check if ˜∆ x k is a descent direction of ϕ π .
Revised: 1 ) , when we check if ˜∆ x k is a descent direction of ϕ π .

Original: Termination Test 1 , Termination Test 2 , Model
Revised: Termination Test 1 , Termination Test 2 , Model

Original: Reduction Condition , and π trial k are referred to in [4].
Revised: Reduction Condition , and π trial k are referred to in [4].

Original: Algorithm 2: [4] with ℓ 1 penalized merit function
Revised: Algorithm 2: [4] with ℓ 1 penalized merit function

Original: Input: Initial iterate ( x 0 , λ 0 ) ; Scalars κ 1 , ϵ, τ, σ, η ∈ (0 , 1) ; ξ B ∈ (0 , 1] ; π 0 , β, κ, κ 2 > 0 ; for k = 0 , 1 , 2 , . . .
Revised: Input: Initial iterate ( x 0 , λ 0 ) ; Scalars κ 1 , ϵ, τ, σ, η ∈ (0 , 1) ; ξ B ∈ (0 , 1] ; π 0 , β, κ, κ 2 > 0 ; for k = 0 , 1 , 2 , . . .

Original: do
Revised: do

Original: Compute f k , ∇ f k , c k , G k , H k , and generate B k
Revised: Compute f k , ∇ f k , c k , G k , H k , and generate B k

Original: Set ( ˜∆ x k , ˜∆ λ k ) ← ( 0 , 0 ) and compute r k by (2.4) while Termination Test 1 AND Termination Test 2 are not satisfied do
Revised: Set ( ˜∆ x k , ˜∆ λ k ) ← ( 0 , 0 ) and compute r k by (2.4) while Termination Test 1 AND Termination Test 2 are not satisfied do

Original: Generate S ∼ P , update ( ˜∆ x k , ˜∆ λ k ) by (2.5), and compute r k by (2.4) end if Termination Test 2 is satisfied and Model Reduction Condition does not hold then
Revised: Generate S ∼ P , update ( ˜∆ x k , ˜∆ λ k ) by (2.5), and compute r k by (2.4) end if Termination Test 2 is satisfied and Model Reduction Condition does not hold then

Original: Set π k ← π trial k + 10 −end
Revised: Set π k ← π trial k + 10 −end

Original: Select α k to satisfy (2.10) using backtracking
Revised: Select α k to satisfy (2.10) using backtracking

Original: Update iterate by (2.11)
Revised: Update iterate by (2.11)

Original: Set π k +1 ← π k end
Revised: Set π k +1 ← π k end

Original: Algorithm 3: version of Algorithm
Revised: Algorithm 3: version of Algorithm

Original: Input: Initial iterate ( x 0 , λ 0 ) ; Scalars κ 0 , η ∈ (0 , 1) , ξ B ∈ (0 , 1] , π 0 > 0 ; ν > 1 ; for k = 0 , 1 , 2 , . . .
Revised: Input: Initial iterate ( x 0 , λ 0 ) ; Scalars κ 0 , η ∈ (0 , 1) , ξ B ∈ (0 , 1] , π 0 > 0 ; ν > 1 ; for k = 0 , 1 , 2 , . . .

Original: do
Revised: do

Original: Compute f k , ∇ f k , c k , G k , H k , and generate B k
Revised: Compute f k , ∇ f k , c k , G k , H k , and generate B k

Original: Set ( ˜∆ x k , ˜∆ λ k ) ← ( 0 , 0 ) and compute r k by (2.4) while Termination Test 1 is not satisfied do while ∥
Revised: Set ( ˜∆ x k , ˜∆ λ k ) ← ( 0 , 0 ) and compute r k by (2.4) while Termination Test 1 is not satisfied do while ∥

Original: r k ∥ 1 > κ ∥∇L k ∥ 1 do
Revised: r k ∥ 1 > κ ∥∇L k ∥ 1 do

Original: Generate S ∼ P , update ( ˜∆ x k , ˜∆ λ k ) by (2.5), and compute r k by (2.4) end if Model Reduction Condition does not hold then
Revised: Generate S ∼ P , update ( ˜∆ x k , ˜∆ λ k ) by (2.5), and compute r k by (2.4) end if Model Reduction Condition does not hold then

Original: Set π k ← π k ν and κ k ← κ k /νend end
Revised: Set π k ← π k ν and κ k ← κ k /νend end

Original: Select α k to satisfy (2.10) using backtracking
Revised: Select α k to satisfy (2.10) using backtracking

Original: Update iterate by (2.11)
Revised: Update iterate by (2.11)

Original: Set π k +1 ← π k end
Revised: Set π k +1 ← π k end

Original: Appendix D. Further Experiments Implementation Details.
Revised: Appendix D. Further Experiments Implementation Details.

Original: Algorithm 1: The proposed algorithm.
Revised: Algorithm 1: The proposed algorithm.

Original: The parameters are set as η 1 , 0 = 1 , η 2 , 0 = 0 .
Revised: The parameters are set as η 1 , 0 = 1 , η 2 , 0 = 0 .

Original: 1 , δ 0 = 0 .
Revised: 1 , δ 0 = 0 .

Original: 1 , ξ B = 0 .
Revised: 1 , ξ B = 0 .

Original: 1 , β = 0 .
Revised: 1 , β = 0 .

Original: 1 , ν = 1 . 4 , ρ = 0 .
Revised: 1 , ν = 1 . 4 , ρ = 0 .

Original: 5 , θ k = 1 . 2.
Revised: 5 , θ k = 1 . 2.

Original: Algorithm 2: [4] with the ℓ 1 penalized merit function.
Revised: Algorithm 2: [4] with the ℓ 1 penalized merit function.

Original: We follow the parameter setup used in [4].
Revised: We follow the parameter setup used in [4].

Original: The parameters are set as π 0 = 1 , κ = 1 , κ 1 = 0 .
Revised: The parameters are set as π 0 = 1 , κ = 1 , κ 1 = 0 .

Original: 1 , ϵ = 0 .
Revised: 1 , ϵ = 0 .

Original: 1 , τ = 0 .
Revised: 1 , τ = 0 .

Original: 1 , η = 10 − 8 .
Revised: 1 , η = 10 − 8 .

Original: Likewise, the remaining parameters are set as ξ B = 0 .
Revised: Likewise, the remaining parameters are set as ξ B = 0 .

Original: 1 , σ = τ (1 − ϵ ) , and κ
Revised: 1 , σ = τ (1 − ϵ ) , and κ

Original: 2 = β
Revised: 2 = β

Original: = ∥∇L 0 ∥∥ c 

Revised: = ∥∇L 0 ∥∥ c

Original: ∥ 1 + 1 ∨ 1 . 3.
Revised: ∥ 1 + 1 ∨ 1 . 3.

Original: Algorithm 3: Adaptive version of Algorithm 2.
Revised: Algorithm 3: Adaptive version of Algorithm 2.

Original: The parameters are set as π 0 = 1 , κ 0 = 0 .
Revised: The parameters are set as π 0 = 1 , κ 0 = 0 .

Original: 1 , η = 10 − 8 , ξ B = 0 .
Revised: 1 , η = 10 − 8 , ξ B = 0 .

Original: 1 , β = 0 .
Revised: 1 , β = 0 .

Original: 1 , ν = 1 . 4 .
Revised: 1 , ν = 1 . 4 .

Original: For the Hessian modification, we regularize the Hessian H k by B k
Revised: For the Hessian modification, we regularize the Hessian H k by B k

Original: = H k +( ξ B + ∥
Revised: = H k +( ξ B + ∥

Original: H k ∥ ) I n whenever
Revised: H k ∥ ) I n whenever

Original: H k does not satisfy Assumption 2.
Revised: H k does not satisfy Assumption 2.

Original: The stopping criterion is set as:
Revised: The stopping criterion is set as:

Original: [Equation]
Revised: [Equation]

Original: If the algorithm terminates by the former stopping criterion, we say the algorithm converges, otherwise the latter stopping criterion would be satisfied.
Revised: If the algorithm terminates by the former stopping criterion, we say the algorithm converges, otherwise the latter stopping criterion would be satisfied.

Original: D.1. CUTEst
Revised: D.1. CUTEst

Original: Among all the problems in the CUTEst test set, we selected the problems for which f is not a constant objective with n < 1000 , containing only equality constraints, positive definiteness of G k G Tk at all iterates of all algorithms that we ran.
Revised: Among all the problems in the CUTEst test set, we selected the problems for which f is not a constant objective with n < 1000 , containing only equality constraints, positive definiteness of G k G Tk at all iterates of all algorithms that we ran.

Original: This selection scheme yields a total of 47 problems.
Revised: This selection scheme yields a total of 47 problems.

Original: Throughout the experiments, we use the initial value of primal-dual variables which are provided by the CUTEst package.
Revised: Throughout the experiments, we use the initial value of primal-dual variables which are provided by the CUTEst package.

Original: For each algorithm, we average over 10 independent runs.
Revised: For each algorithm, we average over 10 independent runs.

Original: We compare Algorithms 2 and 3.
Revised: We compare Algorithms 2 and 3.

Original: Remark that Step acceptance condition of Algorithm 1 is similar with the Termination Test 1 in Algorithm 2, in that both conditions require an inexact step ( ˜∆ x k , ˜∆ λ k ) to acheive certain accuracy and ensure a step to be a descent direction of the merit function.
Revised: Remark that Step acceptance condition of Algorithm 1 is similar with the Termination Test 1 in Algorithm 2, in that both conditions require an inexact step ( ˜∆ x k , ˜∆ λ k ) to acheive certain accuracy and ensure a step to be a descent direction of the merit function.

Original: However, Algorithm 1 adaptively controls the accuracy of an inexact solution of (2.3), on the other hand, Algorithm 2 uses a consistent bound to the accuracy throughout all iterations.
Revised: However, Algorithm 1 adaptively controls the accuracy of an inexact solution of (2.3), on the other hand, Algorithm 2 uses a consistent bound to the accuracy throughout all iterations.

Original: Since Algorithm 3 adaptively controls the accuracy of an inexact search direction, but relies on
Revised: Since Algorithm 3 adaptively controls the accuracy of an inexact search direction, but relies on

Original: Termination Test 1 in Algorithm 2, we can view Algorithm 3 as an adaptive version of Algorithm 2.
Revised: Termination Test 1 in Algorithm 2, we can view Algorithm 3 as an adaptive version of Algorithm 2.

Original: We present the comparison between Algorithms 2 and 3 on CUTEst set in Figure 2.
Revised: We present the comparison between Algorithms 2 and 3 on CUTEst set in Figure 2.

Original: From Figure 2, we observe that Algorithm 3 is superior than Algorithm 2 in all three criteria.
Revised: From Figure 2, we observe that Algorithm 3 is superior than Algorithm 2 in all three criteria.

Original: This is because
Revised: This is because

Original: Algorithm 3 adaptively controls the accuracy of the inexact solution and this adaptive scheme yields tighter bounds on the residuals of the iterative solver.
Revised: Algorithm 3 adaptively controls the accuracy of the inexact solution and this adaptive scheme yields tighter bounds on the residuals of the iterative solver.

Original: This results in steeper decrease in the merit function at each iteration and smaller number of outer iterations.
Revised: This results in steeper decrease in the merit function at each iteration and smaller number of outer iterations.

Original: Since both algorithms use the ℓpenalized merit function, both algorithm do not involve gradient and Jacobian evaluations when we find a stepsize α k .
Revised: Since both algorithms use the ℓpenalized merit function, both algorithm do not involve gradient and Jacobian evaluations when we find a stepsize α k .

Original: D.2. Constrained Logistic Regression
Revised: D.2. Constrained Logistic Regression

Original: We consider equality-constrained logistic regression problems of the form
Revised: We consider equality-constrained logistic regression problems of the form

Original: [Equation]
Revised: [Equation]

Original: where X ∈ R N × n is a feature matrix with n feature dimension and N data points, y ∈ {− 1 , 1 } N contains corresponding label data, A ∈ R m × n and b ∈ R m .
Revised: where X ∈ R N × n is a feature matrix with n feature dimension and N data points, y ∈ {− 1 , 1 } N contains corresponding label data, A ∈ R m × n and b ∈ R m .

Original: We follow the experiment details in [1].
Revised: We follow the experiment details in [1].

Original: Among all datasets in the LIBSVM collection, we consider 7 binary classification datasets for which 12 ≤ n ≤ 1000 , 256 ≤ N ≤ 100000 , and positive definiteness of G k G Tk at all iterates of all algorithms we ran.
Revised: Among all datasets in the LIBSVM collection, we consider 7 binary classification datasets for which 12 ≤ n ≤ 1000 , 256 ≤ N ≤ 100000 , and positive definiteness of G k G Tk at all iterates of all algorithms we ran.

Original: For the linear constraints, we fix m = 10 and randomly generate each entry of A and b from a standard normal distribution for each problem.
Revised: For the linear constraints, we fix m = 10 and randomly generate each entry of A and b from a standard normal distribution for each problem.

Original: Combining with the norm constraint, we use total of 11 number of constraints.
Revised: Combining with the norm constraint, we use total of 11 number of constraints.

Original: For all problems and algorithms, we set the initial primal and dual iterates as the vector of all ones.
Revised: For all problems and algorithms, we set the initial primal and dual iterates as the vector of all ones.

Original: For each algorithm, we average overindependent runs.
Revised: For each algorithm, we average overindependent runs.

Original: Details of the datasets are given in Table 1.
Revised: Details of the datasets are given in Table 1.

Original: [Table]
Revised: [Table]

Original: We evaluate Algorithm 1 and Algorithm 3 with three criteria on the LIBSVM datasets.
Revised: We evaluate Algorithm 1 and Algorithm 3 with three criteria on the LIBSVM datasets.

Original: The boxplots for the criteria are shown in Figure 3.
Revised: The boxplots for the criteria are shown in Figure 3.

Original: From Figure 3, we observe that Algorithm 1 outperforms
Revised: From Figure 3, we observe that Algorithm 1 outperforms

Original: Algorithm 3 in terms of the KKT residual and number of objective and constraints evaluations, but
Revised: Algorithm 3 in terms of the KKT residual and number of objective and constraints evaluations, but

Original: Algorithm 3 has lower number of gradient and Jacobian evaluations than Algorithm 1 as we observed in Subsection D.1.
Revised: Algorithm 3 has lower number of gradient and Jacobian evaluations than Algorithm 1 as we observed in Subsection D.1.