Original: D IFFERENTIALLY P RIVATE O PTIMIZATION FOR S MOOTH N ONCONVEX ERM
Revised: D IFFERENTIALLY P RIVATE O PTIMIZATION FOR S MOOTH N ONCONVEX ERM

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We develop simple differentially private optimization algorithms that move along directions of (expected) descent to find  approximate second-order necessary solution for non-convex ERM problems.
Revised: We develop simple differentially private optimization algorithms that move along directions of (expected) descent to find an approximate second-order necessary solution for non-convex ERM.

Original: We use line search, mini-batching, and a two-phase strategy to improve the speed and practicality of the algorithm.
Revised: We use line search, mini-batching, and a two-phase strategy to improve the speed and practicality of the algorithm.

Original: Numerical experiments demonstrate the effectiveness of these approaches.
Revised: Numerical experiments demonstrate the effectiveness of these approaches.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Privacy protection has become a central issue in machine learning algorithms, and differential privacy (Dwork &
Revised: Privacy protection has become a central issue in machine learning algorithms, and differential privacy (Dwork &

Original: Roth, 2014) is a rigorous and popular framework for quantifying privacy.
Revised: Roth, 2014) is a rigorous and popular framework for quantifying privacy.

Original: In our paper, we propose a differentially private optimization algorithm that finds an approximate second-order necessary solution for ERM problems.
Revised: In our paper, we propose a differentially private optimization algorithm that finds an approximate second-order necessary solution for ERM problems.

Original: We proposed several techniques to improve the practical performance of the method, including backtracking line search, mini-batching, and a heuristic to avoid the effects of conservative assumptions made in the analysis.
Revised: We proposed several techniques to improve the practical performance of the method, including backtracking line search, mini-batching, and a heuristic to avoid the effects of conservative assumptions made in the analysis.

Original: For given f : R d → R , consider the following minimization problem,
Revised: For given f : R d → R , consider the following minimization problem,

Original: [Equation]
Revised: [Equation]

Original: We want to find an approximate second-order necessary solution, defined formally as follows.
Revised: We want to find an approximate second-order necessary solution, defined formally as follows.

Original: Definition 1 ( ( ϵ g , ϵ H ) -2NS) .
Revised: Definition 1 ( ( ϵ g , ϵ H ) -2NS) .

Original: For given positive values of ϵ g and ϵ H , We say that w is an ( ϵ g , ϵ H )
Revised: For given positive values of ϵ g and ϵ H , We say that w is an ( ϵ g , ϵ H )

Original: -approximate second-order necessary solution (abbreviated as ( ϵ g , ϵ H ) -2NS)
Revised: -approximate second-order necessary solution (abbreviated as ( ϵ g , ϵ H ) -2NS)

Original: if
Revised: if

Original: [Equation]
Revised: [Equation]

Original: We are mostly interested in the case of ϵ g
Revised: We are mostly interested in the case of ϵ g

Original: = α and ϵ H = √
Revised: = α and ϵ H = √

Original: Mα , that is, we seek an ( α, √ Mα ) -2NS, where M is the
Revised: Mα , that is, we seek an ( α, √ Mα ) -2NS, where M is the

Original: Lipschitz constant for ∇ 2 f .
Revised: Lipschitz constant for ∇ 2 f .

Original: We will focus on the empirical risk minimization (ERM) problem, defined as follows,
Revised: We will focus on the empirical risk minimization (ERM) problem, defined as follows,

Original: Definition 2 (ERM) .
Revised: Definition 2 (ERM) .

Original: Given a dataset D = { x 1 , . . .
Revised: Given a dataset D = { x 1 , . . .

Original: , x n } and a loss function l ( w, x ) , we seek the parameter w ∈ R d that minimizes the empirical risk
Revised: , x n } and a loss function l ( w, x ) , we seek the parameter w ∈ R d that minimizes the empirical risk

Original: [Equation]
Revised: [Equation]

Original: ERM is a classical problem in machine learning that has been studied extensively; see, for example Shalev-Shwartz &
Revised: ERM is a classical problem in machine learning that has been studied extensively; see, for example Shalev-Shwartz &

Original: Ben-David (2014).
Revised: Ben-David (2014).

Original: In this paper, we describe differentially private (DP) techniques for solving ERM.
Revised: In this paper, we describe differentially private (DP) techniques for solving ERM.

Original: Previous research on DP algorithms for ERM and optimization has focused mainly on convex loss functions.
Revised: Previous research on DP algorithms for ERM and optimization has focused mainly on convex loss functions.

Original: Recent research on differentially private algorithms for non-convex ERM (Wang et al., 2018a; Wang & Xu, 2019; Zhang et al., 2017) targets an approximate stationary point, which satisfies only the first condition in (1).
Revised: Recent research on differentially private algorithms for non-convex ERM (Wang et al., 2018a; Wang & Xu, 2019; Zhang et al., 2017) targets an approximate stationary point, which satisfies only the first condition in (1).

Original: Wang & Xu (2021) proposes a trust-region type (DPTR) algorithm that gives an approximate second-order necessary solution for ERM, satisfying both conditions in (1), for particular choices of ϵ g and ϵ H .
Revised: Wang & Xu (2021) proposes a trust-region type (DP-TR) algorithm that gives an approximate second-order necessary solution for ERM, satisfying both conditions in (1), for particular choices of ϵ g and ϵ H .

Original: This work requires the trust-region subproblem to be solved exactly at each iteration, and fixes the radius of the trust region at a small value, akin to a “short step” in a line-search method.
Revised: This work requires the trust-region subproblem to be solved exactly at each iteration, and fixes the radius of the trust region at a small value, akin to a “short step” in a line-search method.

Original: An earlier work (Wang et al., 2019) proposed the DP-GD algorithm, which takes short steps in a noisy gradient direction, then sorts through all the iterates so generated to find one that satisfies second-order necessary conditions.
Revised: An earlier work (Wang et al., 2019) proposed the DP-GD algorithm, which takes short steps in a noisy gradient direction, then sorts through all the iterates so generated to find one that satisfies second-order necessary conditions.

Original: Our work matches the sample complexity bound in DP-GD, which is O (cid:16) d α 2 √ ρ (cid:17) for ρ - z CDP or O (cid:18) d √ ln(1 /δ )
Revised: Our work matches the sample complexity bound in DP-GD, which is O (cid:16) d α 2 √ ρ (cid:17) for ρ - z CDP or O (cid:18) d √ ln(1 /δ )

Original: α 2 ε (cid:19) for ( ε, δ ) -DP for finding an ( α, √ Mα ) -2NS, and has an iteration complexity of O ( α − 2 ) .
Revised: α 2 ε (cid:19) for ( ε, δ ) -DP for finding an ( α, √ Mα ) -2NS, and has an iteration complexity of O ( α − 2 ) .

Original: Our contributions can be summarized as follows.
Revised: Our contributions can be summarized as follows.

Original: • Our algorithm is elementary and is based on a simple (non-private) line-search algorithm for finding an approximate second-order necessary solution.
Revised: • Our algorithm is elementary and is based on a simple (non-private) line-search algorithm for finding an approximate second-order necessary solution.

Original: It evaluates second-order information (a noisy Hessian matrix) only when insufficient progress can be made using first-order (gradient) information alone.
Revised: It evaluates second-order information (a noisy Hessian matrix) only when insufficient progress can be made using first-order (gradient) information alone.

Original: By contrast, DP-GD uses the (noisy) Hessian only for checking the second-order approximate condition, while DPTR requires the noisy Hessian to be calculated at every iteration.
Revised: By contrast, DP-GD uses the (noisy) Hessian only for checking the second-order approximate condition, while DP-TR requires the noisy Hessian to be calculated at every iteration.

Original: • Our algorithm is practical and fast.
Revised: • Our algorithm is practical and fast.

Original: DPTR has a slightly better sample complexity bound than our method, depending on α − 7 / 4 rather than α − 2 .
Revised: DP-TR has a slightly better sample complexity bound than our method, depending on α − 7 / 4 rather than α − 2 .

Original: However, since our analysis is based on the worst case, we can be more aggressive with step sizes (see below).
Revised: However, since our analysis is based on the worst case, we can be more aggressive with step sizes (see below).

Original: DPTR requires solving the trust-region subproblem exactly, which is relatively expensive and unnecessary when the gradient is large enough to take a productive step.
Revised: DP-TR requires solving the trust-region subproblem exactly, which is relatively expensive and unnecessary when the gradient is large enough to take a productive step.

Original: Experiments demonstrate that our algorithm requires fewer iterations than DPTR and does less computation on average at each iteration, and thus runs significantly faster than DPTR.
Revised: Experiments demonstrate that our algorithm requires fewer iterations than DP-TR, does less computation on average at each iteration, and thus runs significantly faster than DP-TR.

Original: Moreover, we note that the mini-batch version of DPTR has a sample complexity O ( α − 2 ) , matching the sample complexity of the mini-batch version of our algorithm.
Revised: Moreover, we note that the mini-batch version of DP-TR has a sample complexity O ( α − 2 ) , matching the sample complexity of the mini-batch version of our algorithm.

Original: • We use line search and mini-batching to accelerate the algorithm.
Revised: • We use line search and mini-batching to accelerate the algorithm.

Original: Differentially private line search algorithms have been proposed by (Chen & Lee, 2020).
Revised: Differentially private line search algorithms have been proposed by (Chen & Lee, 2020).

Original: We use the same sparse vector technique as used by their work, but provide a tighter analysis of the sensitivity of the query for checking sufficient decrease condition.
Revised: We use the same sparse vector technique as used by their work, but provide a tighter analysis of the sensitivity of the query for checking sufficient decrease condition.

Original: In addition, we provide a rigorous analysis of the guaranteed function decrease with high probability.
Revised: In addition, we provide a rigorous analysis of the guaranteed function decrease with high probability.

Original: • To complement our worst-case analysis, we propose a heuristic that can obtain much more rapid convergence while retaining the guarantees provided by the analysis
Revised: • To complement our worst-case analysis, we propose a heuristic that can obtain much more rapid convergence while retaining the guarantees provided by the analysis.

Original: The remainder of the paper is structured as follows.
Revised: The remainder of the paper is structured as follows.

Original: In Section 2, we review basic definitions and properties from differential privacy, and make some assumptions about the function f to be optimized.
Revised: In Section 2, we review basic definitions and properties from differential privacy, and make some assumptions about the function f to be optimized.

Original: In Section 3, we describe our algorithm and its analysis.
Revised: In Section 3, we describe our algorithm and its analysis.

Original: We will discuss the basic short step version of the algorithm in Section 3.1, and an extension to a practical line search method in Section 3.2.
Revised: We will discuss the basic short step version of the algorithm in Section 3.1, and an extension to a practical line search method in Section 3.2.

Original: A mini-batch adaptation of the algorithm is described in Section 3.3.
Revised: A mini-batch adaptation of the algorithm is described in Section 3.3.

Original: In
Revised: In

Original: Section 4, we present experimental results and demonstrate the effectiveness of our algorithms.
Revised: Section 4, we present experimental results and demonstrate the effectiveness of our algorithms.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: We use several variants of DP for the need of our analysis, including ( ε, δ ) -DP (Dwork & Roth, 2014), ( α, ϵ ) -RDP (Mironov, 2017), and z CDP (Bun & Steinke, 2016).
Revised: We use several variants of DP for the need of our analysis, including ( ε, δ ) -DP (Dwork & Roth, 2014), ( α, ϵ ) -RDP (Mironov, 2017), and z CDP (Bun & Steinke, 2016).

Original: We review their definitions and properties in Appendix A.
Revised: We review their definitions and properties in Appendix A.

Original: We make the following assumptions about the smoothness of the objective function f .
Revised: We make the following assumptions about the smoothness of the objective function f .

Original: Assumption 1.
Revised: Assumption 1.

Original: We assume f is lower bounded by f .
Revised: We assume f is lower bounded by f .

Original: Assume further that f is G -smooth and has M -Lipschitz Hessian, that is, for all w 1 , w 2 ∈ dom( f ) ,
Revised: Assume further that f is G -smooth and has M -Lipschitz Hessian, that is, for all w 1 , w 2 ∈ dom( f ) ,

Original: [Equation]
Revised: [Equation]

Original: where ∥ · ∥ denotes the vector 2 -norm and the matrix 2 -norm respectively.
Revised: where ∥ · ∥ denotes the vector 2 -norm and the matrix 2 -norm respectively.

Original: We use this notation throughout the paper.
Revised: We use this notation throughout the paper.

Original: For the ERM version of f (see Definition 2), we make additional assumptions.
Revised: For the ERM version of f (see Definition 2), we make additional assumptions.

Original: Assumption 2.
Revised: Assumption 2.

Original: For the ERM setting, we assume the loss function is l ( w, x ) is G -smooth and has M -Lipschitz Hessian with respect to w .
Revised: For the ERM setting, we assume the loss function is l ( w, x ) is G -smooth and has M -Lipschitz Hessian with respect to w .

Original: Thus L ( w, D ) (the average loss across n samples) is also G -smooth and has M -Lipschitz Hessian with respect to w .
Revised: Thus L ( w, D ) (the average loss across n samples) is also G -smooth and has M -Lipschitz Hessian with respect to w .

Original: In addition, we assume l ( w, x ) has bounded function values, gradients, and Hessians.
Revised: In addition, we assume l ( w, x ) has bounded function values, gradients, and Hessians.

Original: That is, there are constants B , B g , and B H such that for any w, x we have,
Revised: That is, there are constants B , B g , and B H such that for any w, x we have,

Original: [Equation]
Revised: [Equation]

Original: As a consequence, the l 2 sensitivity of L ( w, D ) and ∇ L ( w, D ) is bounded by B/n and 2 B g /n respectively.
Revised: As a consequence, the l 2 sensitivity of L ( w, D ) and ∇ L ( w, D ) is bounded by B/n and 2 B g /n respectively.

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: To simplify notation, we define g ( w ) :
Revised: To simplify notation, we define g ( w ) :

Original: = ∇ f ( w ) and H ( w ) :
Revised: = ∇ f ( w ) and H ( w ) :

Original: = ∇ 2 f ( w ) .
Revised: = ∇ 2 f ( w ) .

Original: From the definition (20) of ℓ 2 -sensitivity, we have that the sensitivities of f , g , and H are
Revised: From the definition (20) of ℓ 2 -sensitivity, we have that the sensitivities of f , g , and H are

Original: [Equation]
Revised: [Equation]

Original:  M AIN R ESULTS
Revised:  M AIN R ESULTS

Original: Our algorithmic starting point is the elementary algorithm described in Wright & Recht (2022, Chapter 3.6) that has convergence guarantees to points that satisfy approximate second-order necessary conditions.
Revised: Our algorithmic starting point is the elementary algorithm described in Wright & Recht (2022, Chapter 3.6) that has convergence guarantees to points that satisfy approximate second-order necessary conditions.

Original: For simplicity, we use the following notation to describe and analyze the method:
Revised: For simplicity, we use the following notation to describe and analyze the method:

Original: [Equation]
Revised: [Equation]

Original: We employ the Gaussian mechanism to perturb gradients and Hessians, and denote
Revised: We employ the Gaussian mechanism to perturb gradients and Hessians, and denote

Original: [Equation]
Revised: [Equation]

Original: where ε k ∼ N (0 , ∆ 2 g σ 2 g I d ) for some chosen parameter σ g and E k is a symmetric matrix in which each entry on and above its diagonal is i.i.d.
Revised: where ε k ∼ N (0 , ∆ 2 g σ 2 g I d ) for some chosen parameter σ g and E k is a symmetric matrix in which each entry on and above its diagonal is i.i.d.

Original: as N (cid:0) 0 , ∆ 2 H σ 2 H (cid:1) , for some chosen value of σ H .
Revised: as N (cid:0) 0 , ∆ 2 H σ 2 H (cid:1) , for some chosen value of σ H .

Original: Let ˜ λ
Revised: Let ˜ λ

Original: k denote the minimum eigenvalue of ˜ H k with ˜ p
Revised: k denote the minimum eigenvalue of ˜ H k with ˜ p

Original: k the corresponding eigenvector, with sign and norm chosen to satisfy ∥ ˜ p
Revised: k the corresponding eigenvector, with sign and norm chosen to satisfy ∥ ˜ p

Original: [Equation]
Revised: [Equation]

Original: ∥ =
Revised: ∥ =

Original: [Equation]
Revised: [Equation]

Original: and (˜ p k
Revised: and (˜ p k

Original: [Equation]
Revised: [Equation]

Original: T ˜ g k ≤ 0 .
Revised: T ˜ g k ≤ 0 .

Original: Algorithm 1 specifies the general form of our optimization algorithm.
Revised: Algorithm 1 specifies the general form of our optimization algorithm.

Original: We will discuss two strategies — a “short step” strategy and one based on backtracking line search — to choose the step sizes γ k,g and γ k,H to be taken along the directions ˜
Revised: We will discuss two strategies — a “short step” strategy and one based on backtracking line search — to choose the step sizes γ k,g and γ k,H to be taken along the directions ˜

Original: g k and ˜ p k , respectively.
Revised: g k and ˜ p k , respectively.

Original: For each variant, we define a quantity MIN_DEC to be the minimum decrease, and use it together with a specific lower bound on f to define an upper bound T of the required number of iterations.
Revised: For each variant, we define a quantity MIN_DEC to be the minimum decrease, and use it together with a specific lower bound on f to define an upper bound T of the required number of iterations.

Original: In each iteration, we take a step in the negative of the perturbed gradient direction ˜ g
Revised: In each iteration, we take a step in the negative of the perturbed gradient direction ˜ g

Original: k
Revised: k

Original: if ∥ ˜ g k ∥ > ϵ g .
Revised: if ∥ ˜ g k ∥ > ϵ g .

Original: Otherwise, we check the minimum eigenvalue ˜ λ
Revised: Otherwise, we check the minimum eigenvalue ˜ λ

Original: k of the perturbed Hessian ˜ H k .
Revised: k of the perturbed Hessian ˜ H k .

Original: If ˜ λ k < − ϵ H , we take a step along the direction ˜ p k .
Revised: If ˜ λ k < − ϵ H , we take a step along the direction ˜ p k .

Original: In the remaining case, we have ∥ ˜ g k ∥ ≤ ϵ g and ˜ λ
Revised: In the remaining case, we have ∥ ˜ g k ∥ ≤ ϵ g and ˜ λ

Original: k ≥ − ϵ H , so the approximate second-order necessary conditions are satisfied and we output the current iterate w k as a 2NS solution.
Revised: k ≥ − ϵ H , so the approximate second-order necessary conditions are satisfied and we output the current iterate w k as a 2NS solution.

Original: The quantities σ f , σ g , σ H determine the amount of noise added to function, gradient, and Hessian evaluations, respectively, with the goal of preserving privacy via Gaussian Mechanism.
Revised: The quantities σ f , σ g , σ H determine the amount of noise added to function, gradient, and Hessian evaluations, respectively, with the goal of preserving privacy via Gaussian Mechanism.

Original: We can target a certain privacy level for the overall algorithm ( ρ in ρ -zCDP, for example), find an upper bound on the number of iterations required by whatever variant of Algorithm 1 we are using, and then choose σ f , σ g , and σ H to ensure this level of privacy.
Revised: We can target a certain privacy level for the overall algorithm ( ρ in ρ -zCDP, for example), find an upper bound on the number of iterations required by whatever variant of Algorithm 1 we are using, and then choose σ f , σ g , and σ H to ensure this level of privacy.

Original: Conversely, we can choose positive values for σ f , σ g , and σ H and then determine what level of privacy can be ensured by this choice.
Revised: Conversely, we can choose positive values for σ f , σ g , and σ H and then determine what level of privacy can be ensured by this choice.

Original: (Note that we can keep track of the “privacy leakage” as the algorithm progresses, leading to the possibility of adaptive schemes for choosing the σ ’s.)
Revised: We can keep track of the privacy leakage as the algorithm progresses, leading to the possibility of adaptive schemes for choosing the σ ’s.)

Original: 3. S HORT S TEP
Revised: 3. S HORT S TEP

Original: In the short step version of the algorithm, we make choices for the step sizes that are independent of k :
Revised: In the short step version of the algorithm, we make choices for the step sizes that are independent of k :

Original: [Equation]
Revised: [Equation]

Original: The choices of MIN_DEC and the noise parameters σ f , σ g , and σ H are discussed in the following results.
Revised: The choices of MIN_DEC and the noise parameters σ f , σ g , and σ H are discussed in the following results.

Original: First, we discuss the privacy guarantee and its relationship to the noise variances and the number of iterations.
Revised: First, we discuss the privacy guarantee and its relationship to the noise variances and the number of iterations.

Original: Theorem 1.
Revised: Theorem 1.

Original: Let the noise variances σ f , σ g , σ H be given.
Revised: Let the noise variances σ f , σ g , σ H be given.

Original: Suppose a run of Algorithm 1 takes k g gradient steps and k H negative curvature steps.
Revised: Suppose a run of Algorithm 1 takes k g gradient steps and k H negative curvature steps.

Original: Then the run is ρ - z CDP where ρ = 12 (cid:16) 1 σ 2 f + k g
Revised: Then the run is ρ - z CDP where ρ = 12 (cid:16) 1 σ 2 f + k g

Original: + k H
Revised: + k H

Original: σ 2 g
Revised: σ 2 g

Original: [Equation]
Revised: [Equation]

Original: Recall that T is the maximum number of iterations defined in (6) .
Revised: Recall that T is the maximum number of iterations defined in (6) .

Original: Let ¯ ρ = 12 (cid:16) 1 σ 2 f
Revised: Let ¯ ρ = 12 (cid:16) 1 σ 2 f

Original: + Tσ 2 g + Tσ 2 H (cid:17) .
Revised: + Tσ 2 g + Tσ 2 H (cid:17) .

Original: We always have ¯ ρ ≥ ρ , so the algorithm is ¯ ρ - z CDP.
Revised: We always have ¯ ρ ≥ ρ , so the algorithm is ¯ ρ - z CDP.

Original: Conversely, for given ρ > 0 and ρ f ∈ (0 , ρ ) , we can choose
Revised: Conversely, for given ρ > 0 and ρ f ∈ (0 , ρ ) , we can choose

Original: [Equation]
Revised: [Equation]

Original: to ensure that the algorithm is ρ - z CDP.
Revised: to ensure that the algorithm is ρ - z CDP.

Original: Algorithm
Revised: Algorithm

Original: Given: minimum decrease per iteration MIN_DEC, tolerances ϵ g and ϵ H , noise parameters σ f , σ g and σ H
Revised: Given: minimum decrease per iteration MIN_DEC, tolerances ϵ g and ϵ H , noise parameters σ f , σ g and σ H

Original: Initialize w 0 and sample z ∼ N (0 , ∆ 2 f σ 2 f )
Revised: Initialize w 0 and sample z ∼ N (0 , ∆ 2 f σ 2 f )

Original: Compute an upper bound of the required number of iterations as follows
Revised: Compute an upper bound of the required number of iterations as follows

Original: [Equation]
Revised: [Equation]

Original: MIN_DEC
Revised: MIN_DEC

Original: [Equation]
Revised: [Equation]

Original: Set σ g and σ H using T (See theorems for details) for k = 1 , 2 , . .
Revised: Set σ g and σ H using T (See theorems for details) for k = 1 , 2 , . .

Original: , T do
Revised: , T do

Original: Sample ε 
Revised: Sample ε k

Revised: [Equation]

Revised: , ∆

Original: [Equation]
Revised: [Equation]

Original: Compute the perturbed gradient ˜ g
Revised: Compute the perturbed gradient ˜ g

Original: k
Revised: k

Original: = g k + ε k
Revised: = g k + ε k

Original: if ∥ ˜ g k ∥ >
Revised: if ∥ ˜ g k ∥ >

Original: ϵ g then
Revised: ϵ g then

Original: Choose γ k,g and set w k +1 ← w k − γ k,g ˜ g
Revised: Choose γ k,g and set w k +1 ← w k − γ k,g ˜ g

Original: k ▷ Gradient step else
Revised: k ▷ Gradient step else

Original: Sample E k such that E k is a d × d symmetric matrix in which each entry on and above its diagonal is i.i.d.
Revised: Sample E k such that E k is a d × d symmetric matrix in which each entry on and above its diagonal is i.i.d.

Original: as
Revised: as

Original: N (cid:0) 0 , ∆ 2 H σ 2 H (cid:1) .
Revised: N (cid:0) 0 , ∆ 2 H σ 2 H (cid:1) .

Original: Compute perturbed Hessian ˜
Revised: Compute perturbed Hessian ˜

Original: H
Revised: H

Original: k
Revised: k

Original: = H k + E k and (˜ λ k , ˜ p
Revised: = H k + E k and (˜ λ k , ˜ p

Original: k )
Revised: k )

Original: if ˜ λ k < − ϵ H then
Revised: if ˜ λ k < − ϵ H then

Original: Choose γ k,H > 0 and set w k +1 ← w k + γ k,H ˜ p
Revised: Choose γ k,H > 0 and set w k +1 ← w k + γ k,H ˜ p

Original: k ▷ Negative curvature step elsereturn w k end if end if end for
Revised: k ▷ Negative curvature step elsereturn w k end if end if end for

Original: Proof.
Revised: Proof.

Original: The proof follows directly from the z CDP guarantee for the Gaussian mechanism combined with postprocessing and composition of z CDP.
Revised: The proof follows directly from the z CDP guarantee for the Gaussian mechanism combined with postprocessing and composition of z CDP.

Original: Remark.
Revised: Remark.

Revised: In our algorithm, the actual noise is scaled by the corresponding sensitivity ∆ defined in (4).

Revised: We do the same for later algorithms.

Original: In practice, we expect most steps to be gradient steps, so ¯ ρ is an overestimate of the actual privacy level ρ .
Revised: In practice, we expect most steps to be gradient steps, so ¯ ρ is an overestimate of the actual privacy level ρ .

Original: We can be more aggressive in choosing the noise variances.
Revised: We can be more aggressive in choosing the noise variances.

Original: We discuss a two-phase approach in Section 3.4.
Revised: We discuss a two-phase approach in Section 3.4.

Original: We now discuss guarantees oef the output of Algorithm 1.
Revised: We now discuss guarantees of the output of Algorithm 1.

Original: First, we analyze MIN_DEC in each short step.
Revised: First, we analyze MIN_DEC in each short step.

Original: Lemma 2.
Revised: Lemma 2.

Original: With the short step size choices (7) , if the noise satisfies the following conditions for some positive constants c , c 1 , and c 2 such that c 1 < 12 and c 
Revised: With the short step size choices (7) , if the noise satisfies the following conditions for some positive constants c , c 1 , and c 2 such that c 1 < 12 and c 2 + c < 13 ,

Original: [Equation]
Revised: [Equation]

Original: then the amount of decrease in each step is at least
Revised: then the amount of decrease in each step is at least

Original: [Equation]
Revised: [Equation]

Original: The true gradient and true minimum eigenvalue of the Hessian satisfy the following,
Revised: The true gradient and true minimum eigenvalue of the Hessian satisfy the following,

Original: [Equation]
Revised: [Equation]

Original: Remark.
Revised: Remark.

Original: The constants c , c 1 , and c 2 in the conditions (9) above control how accurate our noisy gradient and Hessian estimates are.
Revised: The constants c , c 1 , and c 2 in the conditions (9) above control how accurate our noisy gradient and Hessian estimates are.

Original: MIN_DEC is smaller when we choose smaller constants.
Revised: MIN_DEC is smaller when we choose smaller constants.

Original: We will also have a tighter solution as demonstrated in the corollary below.
Revised: We will also have a tighter solution as demonstrated in the corollary below.

Original: However, we need smaller noise to satisfy the conditions (9), which in turn translates to a larger required sample size n for our ERM problem, as we will see in Theorem 4.
Revised: However, we need smaller noise to satisfy the conditions (9), which in turn translates to a larger required sample size n for our ERM problem, as we will see in Theorem 4.

Original: Corollary 3.
Revised: Corollary 3.

Original: Assuming the noise satisfies (9) at each iteration, the short step version (using (7) , (10) ) of the algorithm will output a ((1 + c 1 ) ϵ g , (1 + c ) ϵ H ) -2NS.
Revised: Assuming the noise satisfies (9) at each iteration, the short step version (using (7) , (10) ) of the algorithm will output a ((1 + c 1 ) ϵ g , (1 + c ) ϵ H ) -2NS.

Original: With the results above, we now analyze the guarantees of the fixed step-size algorithm under the ERM setting.
Revised: With the results above, we now analyze the guarantees of the fixed step-size algorithm under the ERM setting.

Original: Theorem 4 (Sample complexity of the short step algorithm) .
Revised: Theorem 4 (Sample complexity of the short step algorithm) .

Original: Consider the ERM setting.
Revised: Consider the ERM setting.

Original: Suppose that the number of samples n satisfies n ≥ n min , where
Revised: Suppose that the number of samples n satisfies n ≥ n min , where

Original: [Equation]
Revised: [Equation]

Original: With probability at least { (1 − ζT )(1 − C exp ( − cCd )) } T where c and C are universal constants in Lemma D.7, the output of the short step version (using (7) , (10) ) of the algorithm is a ((1 + c 1 ) ϵ g , (1 + c ) ϵ H ) -2NS.
Revised: With probability at least { (1 − ζT )(1 − C exp ( − cCd )) } T where c and C are universal constants in Lemma D.7, the output of the short step version (using (7) , (10) ) of the algorithm is a ((1 + c 1 ) ϵ g , (1 + c ) ϵ H ) -2NS.

Original: With the choice of σ ’s in (8) using ρ f = c f ρ for c f ∈ (0 , 1) , hiding logarithmic terms and constants, the asymptotic dependence of n min on ( ϵ g , ϵ H ) , ρ and d , is
Revised: With the choice of σ ’s in (8) using ρ f = c f ρ for c f ∈ (0 , 1) , hiding logarithmic terms and constants, the asymptotic dependence of n min on ( ϵ g , ϵ H ) , ρ and d , is

Original: [Equation]
Revised: [Equation]

Original: When ( ϵ g , ϵ H )
Revised: When ( ϵ g , ϵ H )

Original: =
Revised: =

Original: ( α, √ Mα ) , the dependence simplifies to d √ ρ ˜ O
Revised: ( α, √ Mα ) , the dependence simplifies to d √ ρ ˜ O

Original: ( α − 2 ) .
Revised: ( α − 2 ) .

Original: Remark.
Revised: Remark.

Original: When the conditions (9) do not hold, the algorithm could fail to converge to a ((1 + c 1 ) ϵ g , (1 + c ) ϵ H ) -2NS.
Revised: When the conditions (9) do not hold, the algorithm could fail to converge to a ((1 + c 1 ) ϵ g , (1 + c ) ϵ H ) -2NS.

Original: First, the noise in the perturbed gradient and Hessian can be so large that the step is not a descent direction.
Revised: First, the noise in the perturbed gradient and Hessian can be so large that the step is not a descent direction.

Original: Second, due to the noise, we may terminate early or fail to terminate timely when checking the approximate second-order conditions.
Revised: Second, due to the noise, we may terminate early or fail to terminate timely when checking the approximate second-order conditions.

Original: If the noise is not excessive, the solution is still acceptable since the noisy evaluations satisfy the termination conditions.
Revised: If the noise is not excessive, the solution is still acceptable since the noisy evaluations satisfy the termination conditions.

Original: 3. L INE S EARCH A LGORITHM
Revised: 3. L INE S EARCH A LGORITHM

Original: Instead of using a conservative fixed step size, we can do a line search using backtracking.
Revised: Instead of using a conservative fixed step size, we can do a line search using backtracking.

Original: The backtracking line search requires an initial value γ 0 , a decrease parameter β ∈ (0 , 1) for the step size, and constants c g ∈ (0 , 1 − c 1 ) , c H ∈ (0 , 1 − c − (cid:113) 83 c 2 ) that determine the amount of decrease we need.
Revised: The backtracking line search requires an initial value γ 0 , a decrease parameter β ∈ (0 , 1) for the step size, and constants c g ∈ (0 , 1 − c 1 ) , c H ∈ (0 , 1 − c − (cid:113) 83 c 2 ) that determine the amount of decrease we need.

Original: Each line search tries in succession the values γ 0 , βγ 0 , β 2 γ 0 , . .
Revised: Each line search tries in succession the values γ 0 , βγ 0 , β 2 γ 0 , . .

Original: , until a value is found that satisfies a sufficient decrease condition.
Revised: , until a value is found that satisfies a sufficient decrease condition.

Original: For gradient steps, the condition is
Revised: For gradient steps, the condition is

Original: [Equation]
Revised: [Equation]

Original: ) while for negative curvature steps it is
Revised: ) while for negative curvature steps it is

Original: [Equation]
Revised: [Equation]

Original: To make line search differentially private, we use the sparse vector technique from (Dwork & Roth, 2014).
Revised: To make line search differentially private, we use the sparse vector technique from (Dwork & Roth, 2014).

Original: We define queries according to (SD1) and (SD2):
Revised: We define queries according to (SD1) and (SD2):

Original: [Equation]
Revised: [Equation]

Original: whose nonnegativity is equivalent to each of the sufficient decrease conditions.
Revised: whose nonnegativity is equivalent to each of the sufficient decrease conditions.

Original: Algorithm 2 specifies the differentially private line search algorithm using SVT, which is adapted from
Revised: Algorithm 2 specifies the differentially private line search algorithm using SVT, which is adapted from

Original: AboveThreshold algorithm
Revised: AboveThreshold algorithm

Original: (Dwork & Roth, 2014).
Revised: (Dwork & Roth, 2014).

Original: By satisfying the sufficient decrease conditions, we try to get a more substantial improvement in the function value than for the short step algorithm.
Revised: By satisfying the sufficient decrease conditions, we try to get a more substantial improvement in the function value than for the short step algorithm.

Original: As a fallback strategy, we use step sizes similar to the short step values (differing only by a constant factor) if the line search fails, yielding a similar decrease to the short-step case.
Revised: As a fallback strategy, we use step sizes similar to the short step values (differing only by a constant factor) if the line search fails, yielding a similar decrease to the short-step case.

Original: We state the complete algorithm enhanced with line search in Algorithm 3.
Revised: We state the complete algorithm enhanced with line search in Algorithm 3.

Original: In the algorithm, we compute the fall back step size ¯ γ and use a multiplier b ( b > 1 ) of them as the initial step size b ¯ γ for the line search.
Revised: In the algorithm, we compute the fall back step size ¯ γ and use a multiplier b ( b > 1 ) of them as the initial step size b ¯ γ for the line search.

Original: We compute the query sensitivity ∆ q accordingly and call the private line search subroutine to find a step size γ that satisfies the sufficient decrease conditions.
Revised: We compute the query sensitivity ∆ q accordingly and call the private line search subroutine to find a step size γ that satisfies the sufficient decrease conditions.

Original: We have the following privacy guarantees.
Revised: We have the following privacy guarantees.

Original: Theorem 5.
Revised: Theorem 5.

Original: Suppose that σ f , σ g , σ H , and λ are given.
Revised: Suppose that σ f , σ g , σ H , and λ are given.

Original: Suppose an actual run of the line search algorithm takes k g gradient steps and k H negative curvature steps.
Revised: Suppose an actual run of the line search algorithm takes k g gradient steps and k H negative curvature steps.

Original: The run is ρ - z CDP where ρ = 12 (cid:16) 1 σ 2 f + k g
Revised: The run is ρ - z CDP where ρ = 12 (cid:16) 1 σ 2 f + k g

Original: + k H
Revised: + k H

Original: σ 2 g + k H
Revised: σ 2 g + k H

Original: σ 2 H
Revised: σ 2 H

Original: + k g
Revised: + k g

Original: + k H λ 2 (cid:17) .
Revised: + k H λ 2 (cid:17) .

Original: Given: query q and its sensitivity ∆ q , initial step size multiplier b , fall back step size ¯ γ , decrease parameter β , privacy parameter λ function DP-L INE S EARCH ( q, ∆ q , γ init , ¯ γ, β, λ )
Revised: Given: query q and its sensitivity ∆ q , initial step size multiplier b , fall back step size ¯ γ , decrease parameter β , privacy parameter λ function DP-L INE S EARCH ( q, ∆ q , γ init , ¯ γ, β, λ )

Original: Initialize γ ← γ init .
Revised: Initialize γ ← γ init .

Original: Sample ξ ∼ Lap (2 λ ∆ q ) for i = 1 , 2 , . .
Revised: Sample ξ ∼ Lap (2 λ ∆ q ) for i = 1 , 2 , . .

Original: , i max =
Revised: , i max =

Original: ⌊ log β ¯ γγ init ⌋ + 1 do
Revised: ⌊ log β ¯ γγ init ⌋ + 1 do

Original: Sample ν i ∼ Lap (4 λ ∆ q )
Revised: Sample ν i ∼ Lap (4 λ ∆ q )

Original: Evaluate q i = q ( γ ) and ˜ q i =
Revised: Evaluate q i = q ( γ ) and ˜ q i =

Original: q i
Revised: q i

Original: + ν
Revised: + ν

Original: i
Revised: i

Original: if ˜ q i ≥ ξ then
Revised: if ˜ q i ≥ ξ then

Original: HALT and output γ else
Revised: HALT and output γ else

Original: Set γ ← βγ end if end for
Revised: Set γ ← βγ end if end for

Original: HALT and output ¯ γ .
Revised: HALT and output ¯ γ .

Original: end function
Revised: end function

Original: Given: noise bound parameters c 1 , c 2 , c , sufficient decrease parameters c g , c H and initial step size multipliers b g , b H , line search decreasing parameters β g , β H , tolerances ϵ g and ϵ H , noise parameters σ f , σ g , σ H and λ SV T
Revised: Given: noise bound parameters c 1 , c 2 , c , sufficient decrease parameters c g , c H and initial step size multipliers b g , b H , line search decreasing parameters β g , β H , tolerances ϵ g and ϵ H , noise parameters σ f , σ g , σ H and λ SV T

Original: Initialize w 0 , sample z ∼ N (0 , ∆ 2 f σ 2 f ) and compute MIN_DEC according to (17)
Revised: Initialize w 0 , sample z ∼ N (0 , ∆ 2 f σ 2 f ) and compute MIN_DEC according to (17)

Original: Compute an upper bound of the required number of iterations T = (cid:108) f ( w 0 )+ | z |− f MIN_DEC (cid:109)
Revised: Compute an upper bound of the required number of iterations T = (cid:108) f ( w 0 )+ | z |− f MIN_DEC (cid:109)

Original: Set ¯ γ g ← 2 (1 − c 1 − c g ) /G
Revised: Set ¯ γ g ← 2 (1 − c 1 − c g ) /G

Original: for k ← 1 , 2 , . .
Revised: for k ← 1 , 2 , . .

Original: , T do
Revised: , T do

Original: Sample ε k ∼ N (cid:0) 0 , ∆ 2 g σ 2 g I d (cid:1)
Revised: Sample ε k ∼ N (cid:0) 0 , ∆ 2 g σ 2 g I d (cid:1)

Original: Compute the perturbed gradient ˜ g
Revised: Compute the perturbed gradient ˜ g

Original: k
Revised: k

Original: = g k + ε k
Revised: = g k + ε k

Original: if ∥ ˜ g k ∥ >
Revised: if ∥ ˜ g k ∥ >

Original: ϵ g then
Revised: ϵ g then

Original: Define q k,g ( γ ) =
Revised: Define q k,g ( γ ) =

Original: f ( w k ) − f ( w k − γ ˜ g k ) − c g γ ∥
Revised: f ( w k ) − f ( w k − γ ˜ g k ) − c g γ ∥

Original: ˜ g k ∥
Revised: ˜ g k ∥

Original: Set γ init k,g ← b g ¯ γ g , ∆ q k,g ← 2 n γ init
Revised: Set γ init k,g ← b g ¯ γ g , ∆ q k,g ← 2 n γ init

Original: k,g B g ∥
Revised: k,g B g ∥

Original: ˜
Revised: ˜

Original: g k
Revised: g k

Original: ∥
Revised: ∥

Original: ▷ Line search query sensitivity γ k,g ← DP-L INE S EARCH ( q k,g , ∆ q k,g , γ init k,g , ¯ γ g , β g , λ SV T )
Revised: ▷ Line search query sensitivity γ k,g ← DP-L INE S EARCH ( q k,g , ∆ q k,g , γ init k,g , ¯ γ g , β g , λ SV T )

Original: ▷ Backtracking line search w k +1 ← w k − γ k,g ˜ g
Revised: ▷ Backtracking line search w k +1 ← w k − γ k,g ˜ g

Original: k ▷ Gradient step else
Revised: k ▷ Gradient step else

Original: Sample E k such that E k is a d × d symmetric matrix in which each entry on and above its diagonal is i.i.d.
Revised: Sample E k such that E k is a d × d symmetric matrix in which each entry on and above its diagonal is i.i.d.

Original: as
Revised: as

Original: N (cid:0) 0 , ∆ 2 H σ 2 H (cid:1) .
Revised: N (cid:0) 0 , ∆ 2 H σ 2 H (cid:1) .

Original: Compute perturbed Hessian ˜
Revised: Compute perturbed Hessian ˜

Original: H
Revised: H

Original: k
Revised: k

Original: = H k + E k
Revised: = H k + E k

Original: Compute the minimum eigenvalue of ˜ H k and the corresponding eigenvector (˜ λ k , ˜ p
Revised: Compute the minimum eigenvalue of ˜ H k and the corresponding eigenvector (˜ λ k , ˜ p

Original: k ) satisfying (5).
Revised: k ) satisfying (5).

Original: if ˜ λ k < − ϵ H then
Revised: if ˜ λ k < − ϵ H then

Original: Define q k,H ( γ ) =
Revised: Define q k,H ( γ ) =

Original: f ( w k ) − f ( w k + γ ˜ p k ) − 12 c
Revised: f ( w k ) − f ( w k + γ ˜ p k ) − 12 c

Original: H γ 2 | ˜ λ k |
Revised: H γ 2 | ˜ λ k |

Original: Set ¯ γ k,H ← t 2 | ˜ λ k | /M , γ init k,H ← b H ¯ γ k,H , ∆ q k,H ← 2 n γ init k,H B g γ k,H ← DP-L INE S EARCH ( q H , ∆ q kH , γ init k,H , ¯ γ k,H , β H , λ SV T )
Revised: Set ¯ γ k,H ← t 2 | ˜ λ k | /M , γ init k,H ← b H ¯ γ k,H , ∆ q k,H ← 2 n γ init k,H B g γ k,H ← DP-L INE S EARCH ( q H , ∆ q kH , γ init k,H , ¯ γ k,H , β H , λ SV T )

Original: ▷ Backtracking line search w k +1 ← w k + γ k,H ˜ p
Revised: ▷ Backtracking line search w k +1 ← w k + γ k,H ˜ p

Original: k ▷ Negative curvature step elsereturn w k end if end if end for
Revised: k ▷ Negative curvature step elsereturn w k end if end if end for

Original: Recall that T is the maximum number of iterations defined in (6) .
Revised: Recall that T is the maximum number of iterations defined in (6) .

Original: Let ¯ ρ = 12 (cid:16) 1 σ 2 f
Revised: Let ¯ ρ = 12 (cid:16) 1 σ 2 f

Original: + Tσ 2 g + Tσ 2 H + Tλ 2 (cid:17) .
Revised: + Tσ 2 g + Tσ 2 H + Tλ 2 (cid:17) .

Original: We always have ¯ ρ ≥ ρ , so the algorithm is ¯ ρ - z CDP.
Revised: We always have ¯ ρ ≥ ρ , so the algorithm is ¯ ρ - z CDP.

Original: Conversely, for given ρ > 0 and ρ f ∈ (0 , ρ ) , we can choose
Revised: Conversely, for given ρ > 0 and ρ f ∈ (0 , ρ ) , we can choose

Original: [Equation]
Revised: [Equation]

Original: to ensure that algorithm is ρ - z CDP.
Revised: to ensure that algorithm is ρ - z CDP.

Original: Proof.
Revised: Proof.

Original: We know that SVT is (1 /λ ) -DP.
Revised: We know that SVT is (1 /λ ) -DP.

Original: Thus, it satisfies (1 / (2 λ 2 )) - z CDP.
Revised: Thus, it satisfies (1 / (2 λ 2 )) - z CDP.

Original: The result follows directly from the z CDP guarantee for the Gaussian mechanism combined with postprocessing and composition of z CDP.
Revised: The result follows directly from the z CDP guarantee for the Gaussian mechanism combined with postprocessing and composition of z CDP.

Original: We now discuss the guarantee of the output of the algorithm.
Revised: We now discuss the guarantee of the output of the algorithm.

Original: We first derive necessary conditions for sufficient decrease.
Revised: We first derive necessary conditions for sufficient decrease.

Original: Lemma 6.
Revised: Lemma 6.

Original: Assume the same bounded noise conditions (9) as before.
Revised: Assume the same bounded noise conditions (9) as before.

Original: With the choice of sufficient decrease coefficients c g ∈ (0 , 1 − c 1 ) , c H ∈ (0 , 1 − c − (cid:113) 83 c 2 ) , let ¯
Revised: With the choice of sufficient decrease coefficients c g ∈ (0 , 1 − c 1 ) , c H ∈ (0 , 1 − c − (cid:113) 83 c 2 ) , let ¯

Original: γ
Revised: γ

Original: g
Revised: g

Original: = 2 (1 − c 1 − c g ) /G
Revised: = 2 (1 − c 1 − c g ) /G

Original: and ¯ γ H
Revised: and ¯ γ H

Original: =
Revised: =

Original: t 2 | ˜ λ | /M as defined in Algorithm 3, the sufficient decrease conditions (SD1) and (SD2) are satisfied when γ ≤ ¯ γ g and γ ∈
Revised: t 2 | ˜ λ | /M as defined in Algorithm 3, the sufficient decrease conditions (SD1) and (SD2) are satisfied when γ ≤ ¯ γ g and γ ∈

Original: [( t 1 /t 2 )¯ γ H , ¯ γ H ] , respectively, where 0 < t 1 < t 2 are solutions to the following quadratic equation (given our choice of c, c 2 , c H , real solutions exist),
Revised: [( t 1 /t 2 )¯ γ H , ¯ γ H ] , respectively, where 0 < t 1 < t 2 are solutions to the following quadratic equation (given our choice of c, c 2 , c H , real solutions exist),

Original: [Equation]
Revised: [Equation]

Original: Explicitly, we have
Revised: Explicitly, we have

Original: [Equation]
Revised: [Equation]

Original: In particular, we have q g (¯ γ g ) ≥ 0 and q H (¯ γ H ) ≥ 0 .
Revised: In particular, we have q g (¯ γ g ) ≥ 0 and q H (¯ γ H ) ≥ 0 .

Original: We now derive the minimum amount of decrease for each iteration.
Revised: We now derive the minimum amount of decrease for each iteration.

Original: Lemma 7.
Revised: Lemma 7.

Original: Using DP line search Algorithm 3, assume the same bounded noise conditions (9) as before.
Revised: Using DP line search Algorithm 3, assume the same bounded noise conditions (9) as before.

Original: With the choice of sufficient decrease coefficients c g ∈ (0 , 1 − c 1 ) , c H ∈ (0 , 1 − c − (cid:113) 83 c 2 ) , define ¯ γ g and ¯ γ H as before.
Revised: With the choice of sufficient decrease coefficients c g ∈ (0 , 1 − c 1 ) , c H ∈ (0 , 1 − c − (cid:113) 83 c 2 ) , define ¯ γ g and ¯ γ H as before.

Original: Choose initial step size multipliers b g , b H > 1 and decrease parameters β g ∈ (0 , 1) , β H ∈ ( t 1 /t 2 , 1) .
Revised: Choose initial step size multipliers b g , b H > 1 and decrease parameters β g ∈ (0 , 1) , β H ∈ ( t 1 /t 2 , 1) .

Original: Let i max =
Revised: Let i max =

Original: ⌊ log β max( b g , b H ) ⌋ + 1 .
Revised: ⌊ log β max( b g , b H ) ⌋ + 1 .

Original: If n satisfies the following:
Revised: If n satisfies the following:

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − ξ/T , the amount of decrease in a single step is at least
Revised: with probability at least 1 − ξ/T , the amount of decrease in a single step is at least

Original: [Equation]
Revised: [Equation]

Original: With the results above, we can now analyze the guarantees of the line search algorithm under ERM settings.
Revised: With the results above, we can now analyze the guarantees of the line search algorithm under ERM settings.

Original: Theorem 8 (Sample complexity of the line search algorithm) .
Revised: Theorem 8 (Sample complexity of the line search algorithm) .

Original: Assuming the same conditions as in the previous lemma, with probability at least { (1 − ζT )(1 − C exp ( − cCd ))
Revised: Assuming the same conditions as in the previous lemma, with probability at least { (1 − ζT )(1 − C exp ( − cCd ))

Original: (1 − ξ/T ) } T , suppose the number of samples n satisfies n ≥ n min , where
Revised: (1 − ξ/T ) } T , suppose the number of samples n satisfies n ≥ n min , where

Original: [Equation]
Revised: [Equation]

Original: The output of the algorithm is a ((1 + c 1 ) ϵ g , (1 + c ) ϵ H ) -2NS.
Revised: The output of the algorithm is a ((1 + c 1 ) ϵ g , (1 + c ) ϵ H ) -2NS.

Original: With the choice of σ ’s and λ in (14) , hiding logarithmic terms and constants, the asymptotic dependence of n min on ( ϵ g , ϵ H ) and ρ , is
Revised: With the choice of σ ’s and λ in (14) , hiding logarithmic terms and constants, the asymptotic dependence of n min on ( ϵ g , ϵ H ) and ρ , is

Original: [Equation]
Revised: [Equation]

Original: When ( ϵ g , ϵ H )
Revised: When ( ϵ g , ϵ H )

Original: =
Revised: =

Original: ( α, √ Mα ) , the dependence simplifies to d √ ρ ˜ O
Revised: ( α, √ Mα ) , the dependence simplifies to d √ ρ ˜ O

Original: ( α − 2 ) .
Revised: ( α − 2 ) .

Original: Proof.
Revised: Proof.

Original: The proof is similar to D.4 using Lemma 17.
Revised: The proof is similar to Theorem 4 (see Appendix D.4) using Lemma 17.

Original: We have an additional term in our success probability due to the SVT line search step.
Revised: We have an additional term in our success probability due to the SVT line search step.

Original: For the asymptotic bound of n min , we note that MIN_DEC (17), T , σ g and σ H are the same as those of the short step algorithm, up to a constant.
Revised: For the asymptotic bound of n min , we note that MIN_DEC (17), T , σ g and σ H are the same as those of the short step algorithm, up to a constant.

Original: The additional requirement (16) for n isO (cid:16)
Revised: The additional requirement (16) for n is

Original: [Equation] (cid:17) .
Revised: O (cid:16) λ log T max( ϵ − 1 g ,ϵ − 2 H ) (cid:17) .

Original: Since we choose λ = σ g , it is in the same order as the first term inside the max expression of n min in (18).
Revised: Since we choose λ = σ g , it is in the same order as the first term inside the max expression of n min in (18).

Original: Thus, the asymptotic bound of n min is the same as that of the short step algorithm.
Revised: Thus, the asymptotic bound of n min is the same as that of the short step algorithm.

Original: 3. M INI - BATCHING
Revised: 3. M INI - BATCHING

Original: We can use mini-batching to speed up the algorithm.
Revised: We can use mini-batching to speed up the algorithm.

Original: Specifically, in each iteration k , we sample m data points from D without replacement, forming the mini-batch S k .
Revised: Specifically, in each iteration k , we sample m data points from D without replacement, forming the mini-batch S k .

Original: The objective now only computes the average risk over set S k , that is, f k,S k := 1 m (cid:88)
Revised: The objective now only computes the average risk over set S k , that is, f [Equation] S [Equation]

Revised: ℓ

Original: i ∈ S k ℓ ( w, x i ) .
Revised: [Equation] w [Equation] x i ) .

Original: Likewise, we modify all the prior algorithms by evaluating the gradients and Hessians over the mini-batch S k .
Revised: Likewise, we modify all the prior algorithms by evaluating the gradients and Hessians over the mini-batch S k .

Original: We show that the sample complexity of the mini-batch version of the algorithm remains ˜ O
Revised: We show that the sample complexity of the mini-batch version of the algorithm remains ˜ O

Original: ( d √ ln(1 /δ ) εα) when ( ϵ g , ϵ H ) =
Revised: ( d √ ln(1 /δ ) εα) when ( ϵ g , ϵ H ) =

Original: ( α, √ Mα ) for ( ε, δ ) -DP.
Revised: ( α, √ Mα ) for ( ε, δ ) -DP.

Original: The details are in the Appendix B.
Revised: The details are in the Appendix B.

Original: 3. D ISCUSSION : P RACTICAL I MPROVEMENT AND E IGENVALUE C OMPUTATION
Revised: 3. D ISCUSSION : P RACTICAL I MPROVEMENT AND E IGENVALUE C OMPUTATION

Original: The estimate of T in (6) is pessimistic, based on MIN_DESC obtained from our worst-case analysis.
Revised: The estimate of T in (6) is pessimistic, based on MIN_DEC obtained from our worst-case analysis.

Original: In practice, we can employ a two-phase strategy to speed up the algorithm.
Revised: we now propose a two-phase strategy to speed up the algorithm.

Original: In the first phase, we use a fraction (say 3 / 4 ) of the privacy budget to try out a smaller T , which corresponds to less noise and quicker convergence.
Revised: In the first phase, we use a fraction (say 3 / 4 ) of the privacy budget to try out a smaller T , which corresponds to less noise and quicker convergence.

Original: If we are unable to find a desired solution, we fall back to the original algorithm with the remaining privacy budget and use the last iterate as a warm start.
Revised: If we are unable to find a desired solution, we fall back to the original algorithm with the remaining privacy budget and use the last iterate as a warm start.

Original: In an actual implementation, we can use Lanczos method to compute the minimum eigenvalue and eigenvector.
Revised: In the actual implementation, we can use Lanczos method to compute the minimum eigenvalue and eigenvector.

Original: This will slightly change our analysis.
Revised: This will slightly change our analysis.

Original: See Appendix C for a discussion.
Revised: See Appendix C for a discussion.

Revised: We comment that Lanczos method only requires Hessianvector product and thus the full Hessian is not required.

Revised: We can use automatic differentiation or finite differencing to obtain Hessian-vector product efficiently, reducing the time complexity of this step to O ( d ) from O ( d 2 ) .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We carry out numerical experiments to demonstrate the performance of our DP optimization algorithms, following similar experimental protocols to (Wang & Xu, 2021).
Revised: We carry out numerical experiments to demonstrate the performance of our DP optimization algorithms, following similar experimental protocols to (Wang & Xu, 2021).

Original: We use Covtype dataset, which has n = 581012 data points.
Revised: We use Covertype dataset and perform necessary data preprocessing.

Original: Each data point has the form ( w, y ) , where w is a 54-dimensional feature vector (first 10 are dimensions numerical, column 11 – 14 is the WildernessArea one-hot vector, and last 40 columns is the SoilType one-hot vector), and y being the label, is one of{ 1 , 2 , . . . For preprocessing, we normalize the first 10 numerical columns, and keep only those samples for which y = 1 , 2 . The number of samples remaining in this restricted set is n = 495141 .
Revised: Details of the dataset and additional experiments can be found in the Appendix E.

Original: We recode y = 2 to y = − 1 so that y  ∈ {− 1 , 1 } .
Revised: Let x i be the feature vector and y i ∈ {− 1 , +1 } be the binary label.

Original: We investigate the non-convex ERM loss:
Revised: We investigate the non-convex ERM loss 2 :

Original: [Equation]
Revised: [Equation]

Original: , where r ( w ) = (cid:80) pi
Revised: where r ( w ) = (cid:80) pi

Original: =1 λw 2 i 1+ w 2 i is the non-convex regularizer.
Revised: =1 λw 2 i 1+ w 2 i is the non-convex regularizer.

Original: 4. R ESULTS
Revised: In our experiments, we choose λ = 10 − 3 .

Original: We implement our algorithms and DPTR in Python using PyTorch.
Revised: We implement our algorithms and DP-TR in Python using PyTorch.

Original:  To make the results comparable, we modify DPTR to explicitly check approximate second-order necessary conditions and stop if these conditions are satisfied, in the same way as in our algorithms.
Revised: 3 To make the results comparable, we modify DP-TR so that it explicitly checks approximate second-order necessary conditions and stop if these conditions are satisfied, in the same way as in our algorithms.

Original: The experiments were run on a cluster with a 36-core Intel Xeon Gold 6254 3.1GHz CPU, utilizing 8 CPU cores for each run.

Original: We run the experiment under two settings:
Revised: We run the experiment under two settings:

Original: Easy case: ϵ g = 0 .
Revised: Finding a loose solution: ϵ g = 0 .

Original: 06 and ϵ H= √ 0 . 06 ≈ 0 .
Revised: 060 and ϵ H ≈ 0 .

Original: In this setting, our requirement on the 2NS is loose.
Revised: In this setting, our requirement for the 2NS is loose.

Original: Thisin turn translates to a large sample size n compared to the required sample complexity.
Revised: This translates to a large sample size n compared to the required sample complexity.

Original: Hard case: ϵ g = 0 .
Revised: Finding a tight solution: ϵ g = 0 .

Original: 03 and ϵ H= √ 0 .
Revised: 030 and ϵ H

Original: 03 ≈ 0 .
Revised: ≈ 0 .

Original: In this setting, our requirement on the 2NS is tight.
Revised: In this setting, our requirement for the 2NS is tight.

Original: We have a small sample size n compared to the required sample complexity.
Revised: We have a small sample size n compared to the required sample complexity.

Original: For each setting, we pick different level of privacy budget ε and run each configuration with five different random seeds.
Revised: For each setting, we pick different levels of privacy budget ε and run each configuration with five different random seeds.

Revised: We convert differential privacy schemes to ( ε, δ ) -DP when necessary for the comparison.

Original: We present the  results in the tables below.
Revised: We present the aggregated results in the tables below.

Original: In each entry, we report (loss mean, loss std), whether the algorithm terminates with success, and (runtime mean, runtime std).
Revised: In each entry, we report mean ± standard deviation of the values across five runs. If any of the five runs failed to find a solution, or it found a solution but failed to terminate due to the noise, we report the runtime with × . Here, the runtime is expressed in a unit determined by the Python function time.perf_counter() . In the table, we use acronyms for methods: TR for DP-TR, OPT for our proposed algorithms and 2OPT for their two-phase variants, OPT-LS for our proposed algorithms with line search, and the ones with "-B" use mini-batching.

Original: The time unit is relative.
Revised: support of this in PyTorch. Upon checking, the loss has Lipschitz gradients and Hessians as long as the feature vector x i ’s are bounded. We implement DP-GD but cannot produce practical results using the algorithmic parameters described in the DP-GD paper. [Table]

Original: [Table]
Revised: [Table]

Revised: Experimental results show that for finding a loose solution under high privacy budgets ε = 0 .

Revised: our short step

Original: [Table]
Revised: [Table]

Original: We use TR to denote DPTR, OPT  to denote the short step algorithm, LS todenote the line search is used, 2OPT todenote the two-phase strategy is used, and B for mini-batching.
Revised: outperforms TR, with much less runtime and lower final loss. Under the low privacy budget ε = 0 . 2 , although OPT can fail to terminate with success, we see that the final loss is even lower than TR. The reason is as follows, due to the conservative estimate of the decrease, the per iteration privacy budget is low, so we cannot check 2NS conditions accurately enough due to the noise.

Original: In  the easy case, 2OPT-LS consistently outperforms other algorithms.
Revised: In practice, we can stop early and the solution is still acceptable despite the failure of the termination. Heuristics may be employed to spend extra privacy budget to check 2NS conditions. Line search and mini-batching improve upon the short step algorithm, especially when combined with our two-phase strategy. We remark that similar to OPT, OPT-LS has an even more conservative theoretical minimum decrease.

Original: For the faithful implementation, the  line search version of our algorithm is faster than the short step version for large privacy budget.
Revised: The two-phase strategy, using an aggressive estimate of the decrease, complements line search. We observe that 2OPT-LS performs consistently well across all privacy budget levels and under two settings. Finally, we remark that the number of Hessian evaluations is minimal. See Appendix E.3 for details.

Revised:  C ONCLUSION

Original: In the hard case, TR can fail to converge when ε is low, but the two-phase variant (DPOPT-LS-two-phase) could still converge given that is not too small ( ≥ 0 .
Revised: We develop simple differentially private optimization algorithms based on an elementary algorithm for finding an approximate second-order optimal point of a smooth nonconvex function. The proposed algorithms take noisy gradient steps or negative curvature steps based on a noisy Hessian on nonconvex ERM problems. To obtain a method that is more practical than conservative short-step methods, we employ line searches, mini-batching, and a two-phase strategy. We track privacy leakage using z CDP (RDP for mini-batching).

Original: 6 in this case).
Revised: Our work matches the sample complexity of DP-GD, but with a much simplified analysis. Although DP-TR has a better sample complexity, its mini-batched version has the same complexity as ours. Our algorithms have a significant advantage over DP-TR in terms of runtime. 2OPT-LS, which combines the line search and the two-phase strategy, consistently outperform DP-TR in numerical experiments.

Original: A A Q UICK R EVIEW OF D IFFERENTIAL P RIVACY
Revised: A A Q UICK R EVIEW OF D IFFERENTIAL P RIVACY

Original: Definition A.1 ( ( ε, δ ) -DP
Revised: Definition A.1 ( ( ε, δ ) -DP

Original: (Dwork & Roth, 2014)) .
Revised: (Dwork & Roth, 2014)) .

Original: A randomized algorithm A is ( ε, δ ) -DP
Revised: A randomized algorithm A is ( ε, δ ) -DP

Original: if for all neighboring datasets D, D ′ and for all events S in the output space of A , the following holds:
Revised: if for all neighboring datasets D, D ′ and for all events S in the output space of A , the following holds:

Original: [Equation]
Revised: [Equation]

Original: When δ =  we say A is ε -DP.
Revised: When δ = 0 , we say A is ε -DP.

Original: In DP-ERM we say D ′ is a neighboring dataset of D if they differ on just one data point that is by changing one data point x k in D to x ′ k  we obtain dataset D ′ .
Revised: In DP-ERM, we say D ′ is a neighboring dataset of D if they differ on just one data point, that is, by changing one data point x k in D to x ′ k , we obtain dataset D ′ .

Original: Rényi-DP (RDP) was introduced by Mironov as a relaxation of the original DP.
Revised: Rényi-DP (RDP) was introduced by Mironov as a relaxation of the original DP.

Original: Definition A.2 (Rényi divergence) .
Revised: Definition A.2 (Rényi divergence) .

Original: For two probability distributions P and Q defined over R , the Rényi divergence of order α > 1 is
Revised: For two probability distributions P and Q defined over R , the Rényi divergence of order α > 1 is

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Definition A.3 ( ( α, ϵ ) -RDP
Revised: Definition A.3 ( ( α, ϵ ) -RDP

Original: (Mironov, 2017)) .
Revised: (Mironov, 2017)) .

Original: A randomized algorithm M : D → R is ( α, ϵ ) -RDP if for all neighboring dataset pairs D, D ′ , the following holds
Revised: A randomized algorithm M : D → R is ( α, ϵ ) -RDP if for all neighboring dataset pairs D, D ′ , the following holds

Original: [Equation]
Revised: [Equation]

Original: Another notion of differential privacy is Zero-Concentrated Differential Privacy ( z CDP), which requires a linear bound for the divergence of all orders.
Revised: Another notion of differential privacy is Zero-Concentrated Differential Privacy ( z CDP), which requires a linear bound for the divergence of all orders.

Original: Definition A.4 ( z CDP (Bun & Steinke, 2016)) .
Revised: Definition A.4 ( z CDP (Bun & Steinke, 2016)) .

Original: A randomized algorithm M : D → R satisfies ( ξ, ρ ) - z CDP if for all neighboring dataset pairs D, D ′ and all α ∈ (1 , ∞ ) , the following holds:
Revised: A randomized algorithm M : D → R satisfies ( ξ, ρ ) - z CDP if for all neighboring dataset pairs D, D ′ and all α ∈ (1 , ∞ ) , the following holds:

Original: [Equation]
Revised: [Equation]

Original: Equivalently, a randomized algorithm M satisfies ρ - z CDP if for all α ∈ (1 , ∞ ) , M satisfies ( α, ξ + ρα ) -RDP.
Revised: Equivalently, a randomized algorithm M satisfies ρ - z CDP if for all α ∈ (1 , ∞ ) , M satisfies ( α, ξ + ρα ) -RDP.

Original: If this definition holdsfor ξ= 0 , we use theterm ρ - z CDP instead.
Revised: If this definition holds ξ 0 , we use the instead.

Original: RDP and z CDP have some properties in common (Mironov, 2017; Bun & Steinke, 2016).
Revised: RDP and z CDP have some properties in common (Mironov, 2017; Bun & Steinke, 2016).

Original: Proposition A.1 (Composition of RDP) .
Revised: Proposition A.1 (Composition of RDP) .

Original: Suppose that M 1 : D → R 1 is ( α, ϵ 1 ) -RDP and M 2 : R 1 × D
Revised: Suppose that M 1 : D → R 1 is ( α, ϵ 1 ) -RDP and M 2 : R 1 × D

Original: → R 2 is ( α, ϵ 2 ) -RDP.
Revised: → R 2 is ( α, ϵ 2 ) -RDP.

Original: Then the mechanism defined as ( X, Y ) , where X ∼ M 1 ( D ) and Y ∼ M 2 ( X, D ) is ( α, ϵ 1 + ϵ 2 ) -RDP.
Revised: Then the mechanism defined as ( X, Y ) , where X ∼ M 1 ( D ) and Y ∼ M 2 ( X, D ) is ( α, ϵ 1 + ϵ 2 ) -RDP.

Original: Proposition A.2 (Composition of z CDP) .
Revised: Proposition A.2 (Composition of z CDP) .

Original: Suppose that M 1 : D → R 1 is ρ 1 - z CDP and M 2 : R 1 × D
Revised: Suppose that M 1 : D → R 1 is ρ 1 - z CDP and M 2 : R 1 × D

Original: → R 2 is ρ 2 - z CDP.
Revised: → R 2 is ρ 2 - z CDP.

Original: Then the mechanism defined as ( X, Y ) , where X ∼ M 1 ( D ) and Y ∼ M 2 ( X, D ) is ( ρ 1 + ρ 2 ) - z CDP.
Revised: Then the mechanism defined as ( X, Y ) , where X ∼ M 1 ( D ) and Y ∼ M 2 ( X, D ) is ( ρ 1 + ρ 2 ) - z CDP.

Original: Proposition A.3 (Preservation under Postprocessing) .
Revised: Proposition A.3 (Preservation under Postprocessing) .

Original: Consider the mappings M : D → R and g : R → R ′ .
Revised: Consider the mappings M : D → R and g : R → R ′ .

Original: We observe that D α ( P ∥ Q ) ≥ D α ( g ( P )
Revised: We observe that D α ( P ∥ Q ) ≥ D α ( g ( P )

Original: ∥ g ( Q )) by the analog of the data processing inequality.
Revised: ∥ g ( Q )) by the analog of the data processing inequality.

Original: This shows that if M ( · ) is ( α, ϵ ) − RDP , so is g ( M ( · )) .
Revised: This shows that if M ( · ) is ( α, ϵ ) − RDP , so is g ( M ( · )) .

Original: Similarly, if M ( · ) is ( ρ ) - z CDP, so is g ( M ( · )) .
Revised: Similarly, if M ( · ) is ( ρ ) - z CDP, so is g ( M ( · )) .

Original: We can convert easily from one notion of differential privacy to another.
Revised: We can convert easily from one notion of differential privacy to another.

Original: Proposition A.

Revised: Proposition A.

Original: (RDP to ( ε, δ ) -DP) .
Revised: (RDP to ( ε, δ ) -DP) .

Original: If M is an ( α, ϵ ) -RDP mechanism, then it is (cid:16) ϵ + log1 /δ
Revised: If M is an ( α, ϵ ) -RDP mechanism, then it is (cid:16) ϵ + log1 /δ

Original: α − 1 , δ (cid:17) -DP for any 0 < δ < 1 .
Revised: α − 1 , δ (cid:17) -DP for any 0 < δ < 1 .

Original: Proposition A.5 ( ϵ -DP to z CDP) .
Revised: Proposition A.5 ( ϵ -DP to z CDP) .

Original: If M is an ε -DP mechanism, then it is also ( 12 ε 2 ) -
Revised: If M is an ε -DP mechanism, then it is also ( 12 ε 2 ) -

Original: z CDP.
Revised: z CDP.

Original: Proposition A.6 (  ( ε, δ ) -DP to z CDP) .
Revised: Proposition A.6 ( z CDP to ( ε, δ ) -DP) .

Original: Suppose that M : D → R is ( ξ, ρ ) - z CDP.
Revised: Suppose that M : D → R is ( ξ, ρ ) - z CDP.

Original: Then M is also ( ε, δ ) -DP for all δ > 0 and
Revised: Then M is also ( ε, δ ) -DP for all δ > 0 and

Original: [Equation]
Revised: [Equation]

Original: Thus to achieve a ( ε, δ ) -DP guarantee for given ε and δ , it suffices to satisfy ( ξ, ρ ) - z CDP with
Revised: Thus to achieve a ( ε, δ ) -DP guarantee for given ε and δ , it suffices to satisfy ( ξ, ρ ) - z CDP with

Original: [Equation]
Revised: [Equation]

Original: A common way to achieve differential privacy is to add Gaussian noise to the output.
Revised: A common way to achieve differential privacy is to add Gaussian noise to the output.

Original: Proposition A.7 (Gaussian Mechanism) .
Revised: Proposition A.7 (Gaussian Mechanism) .

Original: Given any function h : X n → R d , the Gaussian Mechanism is defined as:
Revised: Given any function h : X n → R d , the Gaussian Mechanism is defined as:

Original: [Equation]
Revised: [Equation]

Original: where ∆ h denotes the ℓ 2 -sensitivity of the function h , defined as
Revised: where ∆ h denotes the ℓ 2 -sensitivity of the function h , defined as

Original: [Equation]
Revised: [Equation]

Original: The Gaussian Mechanism G σ h satisfies ( α, α/ (2 σ 2 )) -RDP for all α ∈ [1 , ∞ ) and thus also satisfies (1 / 2 σ 2 ) - z CDP.
Revised: The Gaussian Mechanism G σ h satisfies ( α, α/ (2 σ 2 )) -RDP for all α ∈ [1 , ∞ ) and thus also satisfies (1 / 2 σ 2 ) - z CDP.

Original: We defer results for ( ε, δ ) -DP to Appendix B.2.
Revised: We defer results for ( ε, δ ) -DP to Appendix B.2.

Original: B A NALYSIS OF THE M INI -B ATCH A LGORITHMS
Revised: B A NALYSIS OF THE M INI -B ATCH A LGORITHMS

Original: For the line search version of the algorithm, we need additional assumptions if we want to check the sufficient decrease conditions using the mini-batch loss.
Revised: For the line search version of the algorithm, we need additional assumptions if we want to check the sufficient decrease conditions using the mini-batch loss.

Original: For simplicity, we only consider the short version of the algorithm in this section.
Revised: For simplicity, we only consider the short version of the algorithm in this section.

Original: B
Revised: B. RDP A NALYSIS

Original: RDP A NALYSIS

Original: (In this section, we use s = m/n to denote the sample fraction.)
Revised: (In this section, we use s = m/n to denote the sample fraction.)

Original: We evaluate the gradient and the Hessian similarly on the mini-batch S k , which we will write as g k,S k , H k,S k and let ˜ g k,S k , ˜
Revised: We evaluate the gradient and the Hessian similarly on the mini-batch S k , which we will write as g k,S k , H k,S k and let ˜ g k,S k , ˜

Original: H k,S k be their perturbed versions respectively.
Revised: H k,S k be their perturbed versions respectively.

Original: The other parts of the algorithm remain unchanged.
Revised: The other parts of the algorithm remain unchanged.

Original: The sensitivity ∆ f , ∆ g , ∆ H as stated in (4) will be scaled accordingly by replacing n in their denominator by the mini-batch size | m .
Revised: The sensitivity ∆ f , ∆ g , ∆ H as stated in (4) will be scaled accordingly by replacing n in their denominator by the mini-batch size | m .

Original: Let g k and H k be thethe gradient and the Hessian evaluated on the full dataset D .
Revised: Let g k and H k be the gradient and the Hessian evaluated on the full dataset D .

Original: We can decompose the deviation of their noisy approximation as follows,
Revised: We can decompose the deviation of their noisy approximation as follows,

Original: [Equation]
Revised: [Equation]

Original: where the first term in the bound is due to the added Gaussian noise, and the second term is due to subsampling.
Revised: where the first term in the bound is due to the added Gaussian noise, and the second term is due to subsampling.

Original: We will bound two terms separately with high probability.
Revised: We will bound two terms separately with high probability.

Original: We have  following subsampling concentration results from (Kohler & Lucchi, 2017)
Revised: We have the following subsampling concentration results from (Kohler & Lucchi, 2017)

Original: Lemma B.1 (Gradient deviation bound) .
Revised: Lemma B.1 (Gradient deviation bound) .

Original: We have with probability at least 1 − η that
Revised: We have with probability at least 1 − η that

Original: [Equation]
Revised: [Equation]

Original: Lemma B.2 (Hessian deviation bound) .
Revised: Lemma B.2 (Hessian deviation bound) .

Original: We have with probability at least 1 − η that
Revised: We have with probability at least 1 − η that

Original: [Equation]
Revised: [Equation]

Original: | S [Equation] | . For the subsampling error (21), we use Gaussian concentration results described before in 41 to bound the first term, and subsampling results stated above to bound the second term.
Revised: For the subsampling error (21), we use Gaussian concentration results described before in 41 to bound the first term, and subsampling results stated above to bound the second term.

Original: For iteration k , with probability at least (1 − ζT )(1 − C exp ( − cCd ))
Revised: For iteration k , with probability at least (1 − ζT )(1 − C exp ( − cCd ))

Original: (1 − η/T ) 

Revised: (1 − η/T )

Original: , we have that
Revised: , we have that

Original: [Equation]
Revised: [Equation]

Original: It suffices to require that each term in the right hand sides of the bound above is bounded by 1 / 2 of the corresponding term in the right hand sides of (9), so that we can use a similar analysis.
Revised: It suffices to require that each term in the right-hand sides of the bound above is bounded by 1 / 2 of the corresponding term in the right-hand sides of (9), so that we can use a similar analysis.

Original: For deviation due to subsampling we need,
Revised: For deviation due to subsampling we need,

Original: [Equation]
Revised: [Equation]

Original: Rearranging the terms, we have the condition for the size of the mini-batch,
Revised: Rearranging the terms, we have the condition for the size of the mini-batch,

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: The following convergence result is immediate, based on the same analysis as in the full-batch case (cf.
Revised: The following convergence result is immediate, based on the same analysis as in the full-batch case (cf.

Original: Theorem 8).
Revised: Theorem 8).

Original: Theorem B.3.
Revised: Theorem B.3.

Original: With probability at least { (1 − ζ/T )(1 − C exp ( − cCd ))(1 − η/T ) 2 } T , suppose the number of samples n satisfies n ≥ n min , where
Revised: With probability at least { (1 − ζ/T )(1 − C exp ( − cCd ))(1 − η/T ) 2 } T , suppose the number of samples n satisfies n ≥ n min , where

Original: [Equation]
Revised: [Equation]

Original: The output of the mini-batch short step algorithm is a ((1 + c 1 ) ϵ g , (1 + c ) ϵ H ) -2NS.
Revised: The output of the mini-batch short step algorithm is a ((1 + c 1 ) ϵ g , (1 + c ) ϵ H ) -2NS.

Original: With the choice of σ ’s in (8) , hiding logarithmic terms and constants, the asymptotic dependence of n min on ( ϵ g , ϵ H ) and ρ , is
Revised: With the choice of σ ’s in (8) , hiding logarithmic terms and constants, the asymptotic dependence of n min on ( ϵ g , ϵ H ) and ρ , is

Original: [Equation]
Revised: [Equation]

Original: We now discuss privacy guarantees.
Revised: We now discuss privacy guarantees.

Original: It is impossible to deal with subsampling using z -CDP, but under RDP, Wang et al.
Revised: It is impossible to deal with subsampling using z -CDP, but under RDP, Wang et al.

Original: (2018b) provides a generalized analysis for subsampling.
Revised: (2018b) provides a generalized analysis for subsampling.

Original: Theorem B.4 (RDP for Subsampled Mechanisms) .
Revised: Theorem B.4 (RDP for Subsampled Mechanisms) .

Original: Given a dataset of n points drawn from a domain X and a (randomized) mechanism M that takes an input from X m for m ≤ n , let the randomized algorithm M ◦ subsample be defined as: (1) subsample: subsample without replacement m datapoints of the dataset (with sampling fraction s = m/n ), and (2) apply M : a randomized algorithm taking the subsampled dataset as the input.
Revised: Given a dataset of n points drawn from a domain X and a (randomized) mechanism M that takes an input from X m for m ≤ n , let the randomized algorithm M ◦ subsample be defined as (1) subsample: subsample without replacement m datapoints of the dataset (with sampling fraction s = m/n ), and (2) apply M : a randomized algorithm taking the subsampled dataset as the input.

Original: For all integers α ≥ 2 , if M is ( α, ϵ ( α )) -RDP, then this new randomized algorithm M ◦ subsample obeys ( α, ϵ ′ ( α ))
Revised: For all integers α ≥ 2 , if M is ( α, ϵ ( α )) -RDP, then this new randomized algorithm M ◦ subsample obeys ( α, ϵ ′ ( α ))

Original: -RDP where, ϵ ′
Revised: -RDP where, ϵ ′

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: For the Gaussian mechanism, we have
Revised: For the Gaussian mechanism, we have

Original: [Equation]
Revised: [Equation]

Original: so ϵ M ( ∞ ) = ∞ and the bound simplifies toϵ [Equation] ≤ [Equation]
Revised: so ϵ M ( ∞ ) = ∞ and the bound simplifies to

Original: ϵ

Original: [Equation]
Revised: [Equation]

Original: which we denote as ϵ ′N ( α ; σ, s )
Revised: which we denote as ϵ ′N ( α ; σ, s )

Original: When s is small and σ is large, we can discard higher-order terms and write the right hand side as
Revised: When s is small and σ is large, we can discard higher-order terms and write the right-hand side as

Original: [Equation]
Revised: [Equation]

Original: where we use the approximation e t ≈ 1 + t for small t .
Revised: where we use the approximation e t ≈ 1 + t for small t .

Original: Theorem B.5.
Revised: Theorem B.5.

Original: Consider the short step version of the algorithm using subsampling.
Revised: Consider the short step version of the algorithm using subsampling.

Original: Given the choice of σ f , σ g , σ H , λ and sampling fraction s .
Revised: Given the choice of σ f , σ g , σ H , λ and sampling fraction s .

Original: Suppose an actual run of the subsampled algorithm takes k g gradient steps and k H negative curvature steps.
Revised: Suppose an actual run of the subsampled algorithm takes k g gradient steps and k H negative curvature steps.

Original: The run is data-dependent ( α, ϵ ( α ))
Revised: The run is data-dependent ( α, ϵ ( α ))

Original: -RDP where
Revised: -RDP where

Original: [Equation]
Revised: [Equation]

Original: Let
Revised: Let

Original: [Equation]
Revised: [Equation]

Original: We always have ¯ ϵ ( α ) ≥ ϵ ( α ) , so the algorithm is ( α, ¯ ϵ ( α )) -RDP.
Revised: We always have ¯ ϵ ( α ) ≥ ϵ ( α ) , so the algorithm is ( α, ¯ ϵ ( α )) -RDP.

Original: Given the complexity in the subsampled privacy guarantee, ϵ ′N ( α ; σ, s ) , we do not have an explicit formula to set parameters σ f , σ g , σ H .
Revised: Given the complexity in the subsampled privacy guarantee, ϵ ′N ( α ; σ, s ) , we do not have an explicit formula to set parameters σ f , σ g , σ H .

Original: However, given ( ε, δ ) -DP privacy budget, we can optimize the parameters to meet the privacy guarantee.
Revised: However, given ( ε, δ ) -DP privacy budget, we can optimize the parameters to meet the privacy guarantee.

Original: Recall the conversion from ( α, ϵ ( α ))
Revised: Recall the conversion from ( α, ϵ ( α ))

Original: -RDP to ( ε DP , δ DP ) -DP, given δ , we solve
Revised: -RDP to ( ε DP , δ DP ) -DP, given δ , we solve

Original: [Equation]
Revised: [Equation]

Original: So we can optimize the parameters σ f , σ g , σ H , such that the following objective is minimized
Revised: So we can optimize the parameters σ f , σ g , σ H , such that the following objective is minimized

Original: [Equation]
Revised: [Equation]

Original: where ¯ ε DP is the target privacy budget and we replace ¯ ϵ ( · ) with their corresponding versions (29) and
Revised: where ¯ ε DP is the target privacy budget and we replace ¯ ϵ ( · ) with their corresponding versions (29).

Original: B. S AMPLE COMPLEXITY USING ( ε, δ ) -DP
Revised: B. S AMPLE COMPLEXITY USING ( ε, δ ) -DP

Original: Under the ( ε, δ ) -DP scheme, subsampling is easier to deal with and we will derive a sample complexity bound.
Revised: Under the ( ε, δ ) -DP scheme, subsampling is easier to deal with and we will derive a sample complexity bound.

Original: We first introduce some useful results in ( ε, δ ) -DP.
Revised: We first introduce some useful results in ( ε, δ ) -DP.

Original: Proposition B.1 (Composition of ( ε, δ ) -DP) .
Revised: Proposition B.1 (Composition of ( ε, δ ) -DP) .

Original: Suppose that M 1 : D → R 1 is ( ε 1 , δ 1 ) -DP and M 2 : R 1 × D
Revised: Suppose that M 1 : D → R 1 is ( ε 1 , δ 1 ) -DP and M 2 : R 1 × D

Original: → R 2 is ( ε 2 , δ 2 ) -DP.
Revised: → R 2 is ( ε 2 , δ 2 ) -DP.

Original: Then the mechanism defined as ( X, Y ) , where X ∼ M 1 ( D ) and Y ∼ M 2 ( X, D ) is ( ε 1 + ε 2 , δ 1 + δ 2 ) -DP.
Revised: Then the mechanism defined as ( X, Y ) , where X ∼ M 1 ( D ) and Y ∼ M 2 ( X, D ) is ( ε 1 + ε 2 , δ 1 + δ 2 ) -DP.

Original: Definition B.1 (Gaussian Mechanism for ( ε, δ ) -DP) .
Revised: Definition B.1 (Gaussian Mechanism for ( ε, δ ) -DP) .

Original: Given any function h : X n → R d , the Gaussian Mechanism is defined as:
Revised: Given any function h : X n → R d , the Gaussian Mechanism is defined as:

Original: [Equation]
Revised: [Equation]

Original: where ∆ h be the ℓ 2 -sensitivity of the function h and σ ≥√
Revised: where ∆ h be the ℓ 2 -sensitivity of the function h and σ ≥

Original: [Equation]
Revised: [Equation]

Original: Then, the Gaussian  G σ  satisfies ( ϵ, δ ) -differential privacy.
Revised: Then, the Gaussian Mechanism G σ h satisfies ( ϵ, δ ) -differential privacy.

Original: Theorem B.6 (Privacy amplification via subsampling (Balle et al., 2018)) . .
Revised: Theorem B.6 (Privacy amplification via subsampling (Balle et al., 2018)) . .

Original: Given a dataset of n points drawn from a domain X and a (randomized) mechanism M that takes an input from X m for m ≤ n , let the randomized algorithm
Revised: Given a dataset of n points drawn from a domain X and a (randomized) mechanism M that takes an input from X m for m ≤ n , let the randomized algorithm

Original: M ◦ subsample be defined as: (1) subsample: subsample without replacement m datapoints of the dataset (sampling parameter s = m/n ), and (2) apply M : a randomized algorithm taking the subsampled dataset as the input.
Revised: M ◦ subsample be defined as: (1) subsample: subsample without replacement m datapoints of the dataset (sampling parameter s = m/n ), and (2) apply M : a randomized algorithm taking the subsampled dataset as the input.

Original: If M is ( ε, δ ) -DP, then M ◦ subsample is ( ε ′ , δ ′ )
Revised: If M is ( ε, δ ) -DP, then M ◦ subsample is ( ε ′ , δ ′ )

Original: -DP, where ε ′ =
Revised: -DP, where ε ′ =

Original: log (1 + s ( e ε − 1) ≤ s ( e ε − 1)) and δ ′ = sδ .
Revised: log (1 + s ( e ε − 1) ≤ s ( e ε − 1)) and δ ′ = sδ .

Original: Theorem B.7 (Advanced Composition) .
Revised: Theorem B.7 (Advanced Composition) .

Original: For all ε 0 , δ 0 , δ ′ 0 ≥ 0 , the class of ( ε 0 , δ 0 ) -differentially private mechanisms satisfies ( ε, kδ + δ ′ )
Revised: For all ε 0 , δ 0 , δ ′ 0 ≥ 0 , the class of ( ε 0 , δ 0 ) -differentially private mechanisms satisfies ( ε, kδ + δ ′ )

Original: -differential privacy under k -fold adaptive composition for:
Revised: -differential privacy under k -fold adaptive composition for:

Original: [Equation]
Revised: [Equation]

Original: As a corollary, for 0 < ε < 1 , it suffices to choose ϵ 

Revised: As a corollary, for 0 < ε < 1 , it suffices to choose ϵ

Original: =
Revised: =

Original: ε 2 √ 2 k log(1 /δ ′ ) to ensure the composition is ( ε, kδ + δ ′ )
Revised: ε 2 √ 2 k log(1 /δ ′ ) to ensure the composition is ( ε, kδ + δ ′ )

Original: -DP.
Revised: -DP.

Original: In particular, we can in addition choose δ ′ =
Revised: In particular, we can in addition choose δ ′ =

Original: δ/ 2 and δ 

Revised: δ/ 2 and δ

Original: =
Revised: =

Original: δ/ (2 k ) to satisfy ( ε, δ ) -DP.
Revised: δ/ (2 k ) to satisfy ( ε, δ ) -DP.

Original: We have the following privacy guarantee for the algorithm using sampling without replacement,
Revised: We have the following privacy guarantee for the algorithm using sampling without replacement,

Original: Theorem B.8.
Revised: Theorem B.8.

Original: Consider the short step version of the algorithm using subsampling.
Revised: Consider the short step version of the algorithm using subsampling.

Original: Given privacy parameters ε, δ, ε f , δ f ∈ (0 , 1) such that ε f < ε and δ f < δ , subsampling parameter s .
Revised: Given privacy parameters ε, δ, ε f , δ f ∈ (0 , 1) such that ε f < ε and δ f < δ , subsampling parameter s .

Original: Let ε 

Revised: Let ε

Original: = ( ε − ε f ) / (8 s (cid:112) 2 T ln(2 / ( δ − δ f ))) and δ 0 = ( δ − δ f ) / (4 sT ) , where T is estimated as before in (6) .
Revised: = ( ε − ε f ) / (8 s (cid:112) 2 T ln(2 / ( δ − δ f ))) and δ 0 = ( δ − δ f ) / (4 sT ) , where T is estimated as before in (6) .

Original: Set σ f = √ 2ln(1 .
Revised: Set σ f = √ 2ln(1 .

Original: 25 /δ f )
Revised: 25 /δ f )

Original: ε f , σ g = σ H = √ 2ln(1 .
Revised: ε f , σ g = σ H = √ 2ln(1 .

Original: 25 /δ 0 )
Revised: 25 /δ 0 )

Original: ϵ .
Revised: ϵ .

Original: The algorithm is ( ε, δ ) -DP.
Revised: The algorithm is ( ε, δ ) -DP.

Original: Proof.
Revised: Proof.

Original: By Gaussian mechanism, the step for estimating T is ( ε f , δ f ) -DP.
Revised: By Gaussian mechanism, the step for estimating T is ( ε f , δ f ) -DP.

Original: It suffices to show the remaining steps are ( ε − ε f , δ − δ f ) .
Revised: It suffices to show the remaining steps are ( ε − ε f , δ − δ f ) .

Original: Using advanced composition, we only need to show that each iteration is (4 sε 0 , 2 sδ 0 ) -DP.
Revised: Using advanced composition, we only need to show that each iteration is (4 sε 0 , 2 sδ 0 ) -DP.

Original: Consider a single iteration without subsampling.
Revised: Consider a single iteration without subsampling.

Original: From the usage of Gaussian mechanism and sparse vector technique, we know that computing the perturbed gradient step and the perturbed Hessian step are both ( ε 0 , δ 0 ) -DP, whereas the backtracking line search step is ε 0 -DP.
Revised: From the usage of Gaussian mechanism and sparse vector technique, we know that computing the perturbed gradient step and the perturbed Hessian step are both ( ε 0 , δ 0 ) -DP, whereas the backtracking line search step is ε 0 -DP.

Original: By composition, we know that the whole iteration is (2 ε 0 , 2 δ 0 ) -DP.
Revised: By composition, we know that the whole iteration is (2 ε 0 , 2 δ 0 ) -DP.

Original: Applying the Privacy Amplification Theorem B.6, we know that each iteration using subsampling is (4 sε 0 , 2 sδ 0 ) -DP.
Revised: Applying the Privacy Amplification Theorem B.6, we know that each iteration using subsampling is (4 sε 0 , 2 sδ 0 ) -DP.

Original: Since (as earlier) we expect most steps to be gradient steps, rather than negative curvature steps, we overestimate the privacy leakage.
Revised: Since (as earlier) we expect most steps to be gradient steps, rather than negative curvature steps, we overestimate the privacy leakage.

Original: Theorem B.9.
Revised: Theorem B.9.

Original: For c f ∈ (0 , 1) , setting ε f = c f ε and δ f = c f δ , under the choice of parameters σ g , σ H in Theorem B.3, the asymptotic dependence of n min in Theorem B.3 on ( ϵ g , ϵ H ) , ( ε, δ ) , , is
Revised: For c f ∈ (0 , 1) , setting ε f = c f ε and δ f = c f δ , under the choice of parameters σ g , σ H in Theorem B.3, the asymptotic dependence of n min in Theorem B.3 on ( ϵ g , ϵ H ) , ( ε, δ ) , , is

Original: [Equation]
Revised: [Equation]

Original: When ( ϵ g , ϵ H )
Revised: When ( ϵ g , ϵ H )

Original: =
Revised: =

Original: ( α, √ Mα ) , the dependence simplifies to ˜ O
Revised: ( α, √ Mα ) , the dependence simplifies to ˜ O

Original: ( d √ 
Revised: ( d √ ln(

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: εα) , matching the result in full-batch version of the algorithm by converting ρ - z CDP to ( ε, δ ) -DP via √ ρ
Revised: εα) , matching the result in full-batch version of the algorithm by converting ρ - z CDP to ( ε, δ ) -DP via √ ρ

Original: = O (ε d
Revised: = O (

Original: [Equation]
Revised: [Equation]

Original: ln(1 /δ ) ) using Proposition A.6.
Revised: 1 /δ ) ) using Proposition A.6.

Original: Proof.
Revised: Proof.

Original: As before, we have √ T
Revised: As before, we have √ T

Original: = O (max( ϵ g , ϵ − 3 / 2 H )) .
Revised: = O (max( ϵ g , ϵ − 3 / 2 H )) .

Original: After simplification, the order of σ g and σ H is
Revised: After simplification, the order of σ g and σ H is

Original: [Equation]
Revised: [Equation]

Original: The asymptotic dependence of n min follows by substituting σ g and σ H into (25).
Revised: The asymptotic dependence of n min follows by substituting σ g and σ H into (25).

Original: C C OMPUTATION OF THE S MALLEST E IGENVALUE U SING L ANCZOS METHOD
Revised: C C OMPUTATION OF THE S MALLEST E IGENVALUE U SING L ANCZOS METHOD

Original: In our algorithms, we need to compute the smallest eigenvalue of the perturbed Hessian.
Revised: In our algorithms, we need to compute the smallest eigenvalue of the perturbed Hessian.

Original: This can be done effectively using the randomized Lanczos algorithm.
Revised: This can be done effectively using the randomized Lanczos algorithm.

Original: We have the following result from (Carmon et al., 2017),
Revised: We have the following result from (Carmon et al., 2018),

Original: Lemma C.1.
Revised: Lemma C.1.

Original: Suppose that the Lanczos method is used to estimate the smallest eigenvalue of H starting with a random vector uniformly generated on the unit sphere, where ∥ H ∥ ≤ M .
Revised: Suppose that the Lanczos method is used to estimate the smallest eigenvalue of H starting with a random vector uniformly generated on the unit sphere, where ∥ H ∥ ≤ M .

Original: For any δ ∈ [0 , 1) , this approach finds the smallest eigenvalue of H to an absolute precision of ϵ/ 2 , together with a corresponding direction v , in at mostmin (cid:40) n
Revised: For any δ ∈ [0 , 1) , this approach finds the smallest eigenvalue of H to an absolute precision of ϵ/ 2 , together with a corresponding direction v , in at most

Original: [Equation]
Revised: [Equation]

Revised: ϵ

Original: [Equation]
Revised: [Equation]

Original: iterations with probability at least 1 − δ .
Revised: iterations with probability at least 1 − δ .

Original: To use Lanczos method in our algorithm, we output an estimate ˜ λ of λ min ( ˜ H ) along with the corresponding eigenvector, provided that ˜ λ ≤ − ϵ H / 2 .
Revised: To use Lanczos method in our algorithm, we output an estimate ˜ λ of λ min ( ˜ H ) along with the corresponding eigenvector, provided that ˜ λ ≤ − ϵ H / 2 .

Original: If ˜ λ > − ϵ H / 2 , we declare that λ min ( ˜ H ) ≥ − ϵ H , with an error probability at most δ L .
Revised: If ˜ λ > − ϵ H / 2 , we declare that λ min ( ˜ H ) ≥ − ϵ H , with an error probability at most δ L .

Original: Our analysis and convergence results still hold with ϵ H / 2 replacing ϵ H and adding the success probability of the Lanczos algorithm 1 − δ L to the product of the success probability in each iteration.
Revised: Our analysis and convergence results still hold with ϵ H / 2 replacing ϵ H and adding the success probability of the Lanczos algorithm 1 − δ L to the product of the success probability in each iteration.

Original: D M ISSING P ROOFS D. P ROOF OF L EMMA
Revised: D M ISSING P ROOFS D. P ROOF OF L EMMA

Original: Lemma D.1.
Revised: Lemma D.1.

Original: With the short step size choices (7) , if the noise satisfies the following conditions for some positive constants c , c 1 , and c 2 such that c 1 < 12 and c 2 + c < 13 ,[Equation] [Equation] 32b
Revised: With the short step size choices (7) , if the noise satisfies the following conditions for some positive constants c , c 1 , and c 2 such that c 1 < 12 and c 2 + c < 13 ,

Original: [Equation]
Revised: [Equation]

Original: then the amount of decrease in each step is at least
Revised: then the amount of decrease in each step is at least

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The true gradient and true minimum eigenvalue of the Hessian satisfy the following,
Revised: The true gradient and true minimum eigenvalue of the Hessian satisfy the following,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We will use the following two standard bounds, which follow from the smoothness assumptions on f :
Revised: We will use the following two standard bounds, which follow from the smoothness assumptions on f :

Original: [Equation]
Revised: [Equation]

Original: For simplicity, we drop the iteration number k in the analysis below.
Revised: For simplicity, we drop the iteration number k in the analysis below.

Original: For gradient steps we have ∥ ˜ g ∥ > ϵ g .
Revised: For gradient steps we have ∥ ˜ g ∥ > ϵ g .

Original: We write g = ˜
Revised: We write g = ˜

Original: g − ε .
Revised: g − ε .

Original: Using ∥ ε ∥ ≤ c 1 ϵ g < c 1 ∥ ˜ g ∥ , it follows from (35) that
Revised: Using ∥ ε ∥ ≤ c 1 ϵ g < c 1 ∥ ˜ g ∥ , it follows from (35) that

Original: [Equation]
Revised: [Equation]

Original: while the true gradient satisfies
Revised: while the true gradient satisfies

Original: [Equation]
Revised: [Equation]

Original: When negative curvature steps are taken, we have ˜ λ <
Revised: When negative curvature steps are taken, we have ˜ λ <

Original: − ϵ H .
Revised: − ϵ H .

Original: By assumption, we have ∥ ε ∥ ≤ c 2 M ϵ 2 H <
Revised: By assumption, we have ∥ ε ∥ ≤ c 2 M ϵ 2 H <

Original: c 2 M | ˜ λ | 2 and ∥
Revised: c 2 M | ˜ λ | 2 and ∥

Original: E ∥ ≤ c ϵ H < c | ˜ λ | .
Revised: E ∥ ≤ c ϵ H < c | ˜ λ | .

Original: Recall the definition (5) of ˜ p
Revised: Recall the definition (5) of ˜ p

Original: and we write g =
Revised: and we write g =

Original: ˜
Revised: ˜

Original: g − ε , ˜ H
Revised: g − ε , ˜ H

Original: =
Revised: =

Original: H − E .
Revised: H − E .

Original: From (36), we have
Revised: From (36), we have

Original: [Equation]
Revised: [Equation]

Original: provided that c 2 + c < 1 / 3 .
Revised: provided that c 2 + c < 1 / 3 .

Original: Let λ denote the minimum eigenvalue of H .
Revised: Let λ denote the minimum eigenvalue of H .

Original: It follows from Weyl’s Inequality that
Revised: It follows from Weyl’s Inequality that

Original: [Equation]
Revised: [Equation]

Original: and thus,
Revised: and thus,

Original: [Equation]
Revised: [Equation]

Original: P ROOF OF L EMMA

Original: Lemma D.2.
Revised: Lemma D.2.

Original: Assume the same bounded noise conditions (9) as before.
Revised: Assume the same bounded noise conditions (9) as before.

Original: With the choice of sufficient decrease coefficients c g ∈ (0 , 1 − c 1 ) , c H ∈ (0 , 1 − c − (cid:113) 83 c 2 ) , let ¯
Revised: With the choice of sufficient decrease coefficients c g ∈ (0 , 1 − c 1 ) , c H ∈ (0 , 1 − c − (cid:113) 83 c 2 ) , let ¯

Original: γ
Revised: γ

Original: g
Revised: g

Original: = 2 (1 − c 1 − c g ) /G
Revised: = 2 (1 − c 1 − c g ) /G

Original: and ¯ γ H
Revised: and ¯ γ H

Original: =
Revised: =

Original: t 2 | ˜ λ | /M as defined in Algorithm 3, the sufficient decrease conditions (SD1) and (SD2) are satisfied when γ ≤ ¯ γ g and γ ∈
Revised: t 2 | ˜ λ | /M as defined in Algorithm 3, the sufficient decrease conditions (SD1) and (SD2) are satisfied when γ ≤ ¯ γ g and γ ∈

Original: [( t 1 /t 2 )¯ γ H , ¯ γ H ] , respectively, where 0 < t 1 < t 2 are solutions to the following quadratic equation (given our choice of c, c 2 , c H , real solutions exist),
Revised: [( t 1 /t 2 )¯ γ H , ¯ γ H ] , respectively, where 0 < t 1 < t 2 are solutions to the following quadratic equation (given our choice of c, c 2 , c H , real solutions exist),

Original: [Equation]
Revised: [Equation]

Original: Explicitly, we have
Revised: Explicitly, we have

Original: [Equation]
Revised: [Equation]

Original: In particular, we have q g (¯ γ g ) ≥ 0 and q H (¯ γ H ) ≥ 0 .
Revised: In particular, we have q g (¯ γ g ) ≥ 0 and q H (¯ γ H ) ≥ 0 .

Original: Proof.
Revised: Proof.

Original: The analysis is similar to that of Lemma 2.
Revised: The analysis is similar to that of Lemma 2.

Original: Again for simplicity we drop iteration indices k .
Revised: Again for simplicity we drop iteration indices k .

Original: For gradient steps we have ∥ ˜ g ∥ > ϵ g .
Revised: For gradient steps we have ∥ ˜ g ∥ > ϵ g .

Original: We write g = ˜
Revised: We write g = ˜

Original: g − ε .
Revised: g − ε .

Original: Using ∥ ε ∥ ≤ c 1 ϵ g < c 1 ∥ ˜ g ∥ , it follows from (35) that
Revised: Using ∥ ε ∥ ≤ c 1 ϵ g < c 1 ∥ ˜ g ∥ , it follows from (35) that

Original: [Equation]
Revised: [Equation]

Original: It follows by definition of ¯ γ g that (SD1) holds when γ ≤ ¯ γ g and
Revised: It follows by definition of ¯ γ g that (SD1) holds when γ ≤ ¯ γ g and

Original: When negative curvature steps are taken, we have ˜ λ <
Revised: When negative curvature steps are taken, we have ˜ λ <

Original: − ϵ H .
Revised: − ϵ H .

Original: By assumption, we have ∥ ε ∥ ≤ c 2 M ϵ 2 H <
Revised: By assumption, we have ∥ ε ∥ ≤ c 2 M ϵ 2 H <

Original: c 2 M | ˜ λ | 2 and ∥
Revised: c 2 M | ˜ λ | 2 and ∥

Original: E ∥ ≤ c ϵ H < c | ˜ λ | .
Revised: E ∥ ≤ c ϵ H < c | ˜ λ | .

Original: Recall the definition (5) of ˜ p
Revised: Recall the definition (5) of ˜ p

Original: and we write g =
Revised: and we write g =

Original: ˜
Revised: ˜

Original: g − ε , ˜ H
Revised: g − ε , ˜ H

Original: =
Revised: =

Original: H − E .
Revised: H − E .

Original: From (36), we have for γ >that
Revised: From (36), we have for γ >that

Original: [Equation]
Revised: [Equation]

Original: By reparameterizing γ = t | ˜ λ | M , we obtain
Revised: By reparameterizing γ = t | ˜ λ | M , we obtain

Original: [Equation]
Revised: [Equation]

Original: Note that (SD2) holds when g ( γ ) ≥
Revised: Note that (SD2) holds when g ( γ ) ≥

Original: 12 c H γ 2 | ˜ λ | .
Revised: 12 c H γ 2 | ˜ λ | .

Original: The result follows from the fact that r ( t ) ≥ 0 for t ∈ [ t 1 , t 2 ] .
Revised: The result follows from the fact that r ( t ) ≥ 0 for t ∈ [ t 1 , t 2 ] .

Original: D. P ROOF OF L EMMA
Revised: D. P ROOF OF L EMMA

Original: Lemma D.3.
Revised: Lemma D.3.

Original: Using DP line search Algorithm 3, assume the same bounded noise conditions (9) as before.
Revised: Using DP line search Algorithm 3, assume the same bounded noise conditions (9) as before.

Original: With the choice of sufficient decrease coefficients c g ∈ (0 , 1 − c 1 ) , c H ∈ (0 , 1 − c − (cid:113) 83 c 2 ) , define ¯ γ g and ¯ γ H as before.
Revised: With the choice of sufficient decrease coefficients c g ∈ (0 , 1 − c 1 ) , c H ∈ (0 , 1 − c − (cid:113) 83 c 2 ) , define ¯ γ g and ¯ γ H as before.

Original: Choose initial step size multipliers b g , b H > 1 and decrease parameters β g ∈ (0 , 1) , β H ∈ ( t 1 /t 2 , 1) .
Revised: Choose initial step size multipliers b g , b H > 1 and decrease parameters β g ∈ (0 , 1) , β H ∈ ( t 1 /t 2 , 1) .

Original: Let i max =
Revised: Let i max =

Original: ⌊ log β max( b g , b H ) ⌋ + 1 .
Revised: ⌊ log β max( b g , b H ) ⌋ + 1 .

Original: If n satisfies the following:
Revised: If n satisfies the following:

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − ξ/T , the amount of decrease in a single step is at least
Revised: with probability at least 1 − ξ/T , the amount of decrease in a single step is at least

Original: MIN_DEC = min

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The analysis is similar to that of Lemma 2.
Revised: The analysis is similar to that of Lemma 2.

Original: Again for simplicity we drop iteration indices k .
Revised: Again for simplicity we drop iteration indices k .

Original: For gradient steps we have ∥ ˜ g ∥ > ϵ g .
Revised: For gradient steps we have ∥ ˜ g ∥ > ϵ g .

Original: We write g = ˜
Revised: We write g = ˜

Original: g − ε .
Revised: g − ε .

Original: Using ∥ ε ∥ ≤ c 1 ϵ g < c 1 ∥ ˜ g ∥ , it follows from (35) that
Revised: Using ∥ ε ∥ ≤ c 1 ϵ g < c 1 ∥ ˜ g ∥ , it follows from (35) that

Original: [Equation]
Revised: [Equation]

Original: It follows by definition of ¯ γ g that (SD1) holds when γ ≤ ¯ γ g and
Revised: It follows by definition of ¯ γ g that (SD1) holds when γ ≤ ¯ γ g and

Original: When negative curvature steps are taken, we have ˜ λ <
Revised: When negative curvature steps are taken, we have ˜ λ <

Original: − ϵ H .
Revised: − ϵ H .

Original: By assumption, we have ∥ ε ∥ ≤ c 2 M ϵ 2 H <
Revised: By assumption, we have ∥ ε ∥ ≤ c 2 M ϵ 2 H <

Original: c 2 M | ˜ λ | 2 and ∥
Revised: c 2 M | ˜ λ | 2 and ∥

Original: E ∥ ≤ c ϵ H < c | ˜ λ | .
Revised: E ∥ ≤ c ϵ H < c | ˜ λ | .

Original: Recall the definition (5) of ˜ p
Revised: Recall the definition (5) of ˜ p

Original: and we write g =
Revised: and we write g =

Original: ˜
Revised: ˜

Original: g − ε , ˜ H
Revised: g − ε , ˜ H

Original: =
Revised: =

Original: H − E .
Revised: H − E .

Original: From (36), we have for γ >that
Revised: From (36), we have for γ >that

Original: [Equation]
Revised: [Equation]

Original: By reparameterizing γ = t | ˜ λ | M , we obtain
Revised: By reparameterizing γ = t | ˜ λ | M , we obtain

Original: [Equation]
Revised: [Equation]

Original: Note that (SD2) holds when g ( γ ) ≥
Revised: Note that (SD2) holds when g ( γ ) ≥

Original: 12 c H γ 2 | ˜ λ | .
Revised: 12 c H γ 2 | ˜ λ | .

Original: The result follows from the fact that r ( t ) ≥ 0 for t ∈ [ t 1 , t 2 ] .
Revised: The result follows from the fact that r ( t ) ≥ 0 for t ∈ [ t 1 , t 2 ] .

Original: D.P ROOF OF C OROLLARY D.
Revised: D.

Revised: P ROOF OF C OROLLARY

Original: Corollary D.4.
Revised: Corollary D.4.

Original:  Assuming the noise satisfies (9) at each iteration, the short step version (using (7) , (10) ) of the algorithm will output a ((1 + c 1 ) ϵ g , (1 + c ) ϵ H ) -2NS.
Revised: 3 Assuming the noise satisfies (9) at each iteration, the short step version (using (7) , (10) ) of the algorithm will output a ((1 + c 1 ) ϵ g , (1 + c ) ϵ H ) -2NS.

Original: Proof.
Revised: Proof.

Original: From the minimum decrease (10) we just derived, it follows that the algorithm will terminate in T ∗ iterations, where
Revised: From the minimum decrease (10) we just derived, it follows that the algorithm will terminate in T ∗ iterations, where

Original: [Equation]
Revised: [Equation]

Original: Our choice of T in (6) is an upper bound of T ∗ and thus the algorithm will halt within T iterations.
Revised: Our choice of T in (6) is an upper bound of T ∗ and thus the algorithm will halt within T iterations.

Original: In the iteration k when the algorithm halts, we have ∥ ˜ g k ∥ ≤ ϵ g and ˜ λ
Revised: In the iteration k when the algorithm halts, we have ∥ ˜ g k ∥ ≤ ϵ g and ˜ λ

Original: k ≥ − ϵ H .
Revised: k ≥ − ϵ H .

Original: It follows from (11) that the output is a ((1+ c 1 ) ϵ g , (1+ c ) ϵ H ) - 2NS.
Revised: It follows from (11) that the output is a ((1+ c 1 ) ϵ g , (1+ c ) ϵ H ) - 2NS.

Original: D. P ROOF OF T HEOREM
Revised: D P ROOF OF T HEOREM

Original: Theorem D.5 (Sample complexity of the short step algorithm) .
Revised: Theorem D.5 (Sample complexity of the short step algorithm) .

Original: Consider the ERM setting.
Revised: Consider the ERM setting.

Original: Suppose that the number of samples n satisfies n ≥ n min , where
Revised: Suppose that the number of samples n satisfies n ≥ n min , where

Original: [Equation]
Revised: [Equation]

Original: With probability at least { (1 − ζT )(1 − C exp ( − cCd )) } T where c and C are universal constants in Lemma D.7, the output of the short step version (using (7) , (10) ) of the algorithm is a ((1 + c 1 ) ϵ g , (1 + c ) ϵ H ) -2NS.
Revised: With probability at least { (1 − ζT )(1 − C exp ( − cCd )) } T where c and C are universal constants in Lemma D.7, the output of the short step version (using (7) , (10) ) of the algorithm is a ((1 + c 1 ) ϵ g , (1 + c ) ϵ H ) -2NS.

Original: With the choice of σ ’s in (8) using ρ f = c f ρ for c f ∈ (0 , 1) , hiding logarithmic terms and constants, the asymptotic dependence of n min on ( ϵ g , ϵ H ) , ρ and d , is
Revised: With the choice of σ ’s in (8) using ρ f = c f ρ for c f ∈ (0 , 1) , hiding logarithmic terms and constants, the asymptotic dependence of n min on ( ϵ g , ϵ H ) , ρ and d , is

Original: [Equation]
Revised: [Equation]

Original: When ( ϵ g , ϵ H )
Revised: When ( ϵ g , ϵ H )

Original: =
Revised: =

Original: ( α, √ Mα ) , the dependence simplifies to d √ ρ ˜ O
Revised: ( α, √ Mα ) , the dependence simplifies to d √ ρ ˜ O

Original: ( α − 2 ) .
Revised: ( α − 2 ) .

Original: Before proving Theorem 4, we introduce two concentration results.
Revised: Before proving Theorem 4, we introduce two concentration results.

Original: Lemma D.6 (Gaussian concentration, (Vershynin, 2018)) .
Revised: Lemma D.6 (Gaussian concentration, (Vershynin, 2018)) .

Original: For x ∼ N (cid:0) 0 , σ 2 I d (cid:1) , with probability at least 1 − η for any 1 > η > 0 , we have
Revised: For x ∼ N (cid:0) 0 , σ 2 I d (cid:1) , with probability at least 1 − η for any 1 > η > 0 , we have

Original: [Equation]
Revised: [Equation]

Original: Lemma D.7 (Upper tail estimate for Wigner ensembles (Tao, 2012, p. 110)) .
Revised: Lemma D.7 (Upper tail estimate for Wigner ensembles (Tao, 2012, p. 110)) .

Original: Let M =
Revised: Let M =

Original: ( m ij ) 1 ≤ i,j ≤ d be an d × d random symmetric matrix.
Revised: ( m ij ) 1 ≤ i,j ≤ d be an d × d random symmetric matrix.

Original: Suppose that the coefficients m ij of M are independent for j ≥ i , mean zero, and have uniform sub-Gaussian tails.
Revised: Suppose that the coefficients m ij of M are independent for j ≥ i , mean zero, and have uniform sub-Gaussian tails.

Original: There exist universal constants C, c > 0 such that for all A ≥ C , we have
Revised: There exist universal constants C, c > 0 such that for all A ≥ C , we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: It follows from concentration results that, in iteration k , with probability at least (1 − ζT )(1 − C [Equation] ( − cCd )) , we have
Revised: It follows from concentration results that, in iteration k , with probability at least (1 − ζT )(1 − C exp ( − cCd )) , we have

Original: [Equation]
Revised: [Equation]

Original: We need to find a condition on n that ensures that the right hand sides are less than the right hand sides of (9).
Revised: We need to find a condition on n that ensures that the right-hand sides are less than the right-hand sides of (9).

Original: We substitute for ∆ g and ∆ H from (4) and solve for n min by rearranging the terms.
Revised: We substitute for ∆ g and ∆ H from (4) and solve for n min by rearranging the terms.

Original: The result then follows from Corollary D.4 if the concentration results hold for all iterations.
Revised: The result then follows from Corollaryif the concentration results hold for all iterations.

Original: Now let us calculate the success probability.
Revised: Now let us calculate the success probability.

Original: For each iteration, we have a probability of at least (1 − ζT )(1 −
Revised: For each iteration, we have a probability of at least (1 − ζT )(1 −

Original: C exp ( − cCd )) that the concentration results hold (if we do not compute the perturbed Hessian, the probability is higher with at least 1 − ζT .
Revised: C exp ( − cCd )) that the concentration results hold (if we do not compute the perturbed Hessian, the probability is higher with at least 1 − ζT .

Original: Using conditional probability, the overall success probability is { (1 − ζT )(1 − C exp ( − cCd )) } τ conditioned on the number of iterations τ .
Revised: Using conditional probability, the overall success probability is { (1 − ζT )(1 − C exp ( − cCd )) } τ conditioned on the number of iterations τ .

Original: Since τ ≤ T , the overall success probability is at least { (1 − ζT )(1 −
Revised: Since τ ≤ T , the overall success probability is at least { (1 − ζT )(1 −

Original: C exp ( − cCd )) } T .
Revised: C exp ( − cCd )) } T .

Original: For the second part, recall from (8) that σ g = σ H = √ T / (cid:112) (1 − c f ) ρ .
Revised: For the second part, recall from (8) that σ g = σ H = √ T / (cid:112) (1 − c f ) ρ .

Original: With (10) and our choice of T in (6), we have √
Revised: With (10) and our choice of T in (6), we have √

Original: T =
Revised: T =

Original: O (max( ϵ g , ϵ − 3 / 2 H )) .
Revised: O (max( ϵ g , ϵ − 3 / 2 H )) .

Original: We obtain the asymptotic bound of n min by plugging in σ g and σ H .
Revised: We obtain the asymptotic bound of n min by plugging in σ g and σ H .

Revised: E E XPERIMENTAL S ETTINGS AND A DDITIONAL E XPERIMENTS

Revised: We first remark that all our experiments were run on a cluster with a 36-core Intel Xeon Gold 6254 3.1GHz CPU, utilizing 8 CPU cores for each run.

Revised: E. D ATASETS

Revised: The Covertype dataset 4 contains n = 581012 data points.

Revised: Each data point has the form ( x, y ) , where x is a 54dimensional feature vector (first 10 are dimensions numerical, column 11 – 14 is the WildernessArea one-hot vector, and last 40 columns are the SoilType one-hot vector), and y being the label, is one of { 1 , 2 , . . .

Revised: /covertype

Revised: For preprocessing, we normalize the first 10 numerical columns, and keep only those samples for which y = 1 , 2 .

Revised: The number of samples remaining in this restricted set is n = 495141 .

Revised: We recode y = 2 to y = − 1 so that y ∈ {− 1 , 1 } .

Revised: The IJCNN dataset 5 contatins n = 4999 data points.

Revised: Each point consists of ( x, y ) , where x is a 22-dimensional feature vector (first 10 are one hot and column 11 – 22 are numerical, and y being the label is binary.

Revised: For preprocessing, we normalize the data.

Revised: For privacy accounting of RDP, which is used in the mini-batched algorithm, we use the autodp package 6 .

Revised: Below we repeat the same experiment using the IJCNN dataset.

Revised: E. IJCNN EXPERIMENT USING LOSS IN S ECTION

Revised: [Table]

Revised: [Table]

Revised: 666 ± 0 .1 .

Revised: 1 ± 0 .2OPT-LS <NA> <NA> 0 .

Revised: 623 ± 0 .× 0 .

Revised: 649 ± 0 .0 .

Revised: 1 ± 0 .

Revised: We remark that for 2OPT-LS under ε = 0 .

Revised: 2 , the result is unavailable (reported as <NA>) because due to numerical issues, the package autodp we use cannot handle subsampling with a very low privacy budget.

Revised: E. A DDITIONAL E XPERIMENTS

Revised: Additionally, we consider the logistic loss

Revised: [Equation]

Revised: and repeat our experiments on the aforementioned datasets with λ = 10 − 3 .

Revised: We can verify that the two chosen losses have Lipschitz gradients and Hessians as long as the feature vector x i ’s are bounded.

Revised: In this set of experiments

Revised: [Table]

Revised: we find solutions ( ϵ g , ϵ H ) =

Revised: 040 , 0 . 200) and ( ϵ g , ϵ H ) =

Revised: We also show the aggregated results for the number of noisy Hessian evaluations.

Revised: We note that the number of noisy Hessian evaluations required in our algorithm is very low, whereas DP-TR needs to evaluate the noisy Hessian every iteration.

Revised: E.3.

Revised: C OVERTYPE EXPERIMENT USING LOGISTIC LOSS

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: E.3. IJCNN EXPERIMENT USING LOGISTIC LOSS

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]