Original: Bring Your Own Algorithm for Optimal Differentially Private Stochastic Minimax Optimization
Revised: Bring Your Own Algorithm for Optimal Differentially Private Stochastic Minimax Optimization

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: Affiliation
Revised: Affiliation

Original: Address email
Revised: Address email

Original: Abstract
Revised: Abstract

Original: We study differentially private (DP) algorithms for smooth stochastic minimaxoptimization, with stochastic minimization as a byproduct.
Revised: We study differentially private (DP) algorithms for smooth stochastic minimaxoptimization, with stochastic minimization as a byproduct.

Original: The holy grail of thesesettings is to guarantee the optimal trade-off between the privacy and the excesspopulation loss, using an algorithm with a linear time-complexity in the number oftraining samples.
Revised: The holy grail of thesesettings is to guarantee the optimal trade-off between the privacy and the excesspopulation loss, using an algorithm with a linear time-complexity in the numberof training samples.

Original: We provide a general framework for solving differentially privatestochastic minimax optimization (DP-SMO) problems, such that practitionerscan bring their own base optimization algorithm and take it as a black-box toobtain the near-optimal privacy-loss trade-off.
Revised: We provide a general framework for solving differentiallyprivate stochastic minimax optimization (DP-SMO) problems, such that practitioners can bring their own base optimization algorithm and take it as a black-box toobtain the near-optimal privacy-loss trade-off.

Original: Our framework is inspired fromthe recently proposed Phased-ERM method [20] for nonsmooth differentiallyprivate stochastic convex optimization (DP-SCO), which exploits the stability ofthe empirical risk minimization (ERM) for the privacy guarantee.
Revised: Our framework is inspired fromthe recently proposed Phased-ERM method [21] for nonsmooth differentially private stochastic convex optimization (DP-SCO), which exploits the stability of theempirical risk minimization (ERM) for the privacy guarantee.

Original: The flexibility ofour approach enables us to sidestep the requirement that the base algorithm needsto have bounded sensitivity, and allows the use of sophisticated variance-reducedaccelerated methods to achieve near-linear time-complexity.
Revised: The flexibility ofour approach enables us to sidestep the requirement that the base algorithm needsto have bounded sensitivity, and allows the use of sophisticated variance-reducedaccelerated methods to achieve near-linear time-complexity.

Original: To the best of ourknowledge, these are the first linear-time optimal algorithms, up to logarithmicfactors, for smooth DP-SMO when the objective is (strongly-)convex–(strongly-)concave.
Revised: To the best of ourknowledge, these are the first near-linear time algorithms with near-optimal guarantees on the population duality gap for smooth DP-SMO, when the objective is(strongly-)convex–(strongly-)concave.

Original: Additionally, based on our flexible framework, we derive a new family ofnear-linear time algorithms for smooth DP-SCO with optimal privacy-loss tradeoffs for a wider range of smoothness parameters compared to previous algorithms.
Revised: Additionally, based on our flexible framework, we enrich the family of near-linear time algorithms for smooth DP-SCO withnear-optimal privacy-loss trade-offs.

Original:  Introduction
Revised:  Introduction

Original: Machine learning models are nowadays trained using large corpora of data samples collected frommany different entities, e.g., from users of a large software service [26].
Revised: Machine learning models are nowadays trained using large corpora of data samples collected frommany different entities, e g., from users of a large software service [27].

Original: However, it has beenempirically shown that these models can be exploited to reveal private information about thesecontributing entities.
Revised: However, it has beenempirically shown that these models can be exploited to reveal private information about thesecontributing entities.

Original: For example, Carlini et al.
Revised: For example, Carlini et al.

Original: [12] attacked the large language model, GPT-2, toreveal hundreds of verbatim text samples used to train these models.
Revised: [12] attacked the large language model, GPT- toreveal hundreds of verbatim text samples used to train these models.

Original: These attacks violate the privacyof the contributing entities, but naturally they expect that no private information about them can berevealed through these models.
Revised: These attacks violate the privacyof the contributing entities, but naturally they expect that no private information about them can berevealed through these models.

Original: Over the last decade, this expectation was even legislated into lawssuch as GDPR in EU [14].
Revised: Over the last decade, this expectation was even legislated into lawssuch as GDPR in EU [14].

Original: There are many mathematical frameworks formalizing this expectationof privacy, but the most widely accepted one is that of Differential Privacy (DP) [17].
Revised: There are many mathematical frameworks formalizing this expectationof privacy, but the most widely accepted one is that of Differential Privacy (DP) [17].

Original: With highprobability, models satisfying DP cannot be attacked by any adversary to identify that a particulartraining sample was used in its training.
Revised: With highprobability, models satisfying DP cannot be attacked by any adversary to identify that a particulartraining sample was used in its training.

Original: Hence, DP provides any entity plausible deniability that theycontributed to the training set.
Revised: Hence, DP provides any entity plausible deniability that theycontributed to the training set.

Original: However, optimization algorithms for training such models under DPrequire careful design choices to ensure privacy while preventing the degradation of convergence speed and data efficiency.
Revised: However, optimization algorithms for training such models under DPrequire careful design choices to ensure privacy while preventing the degradation of convergence speed and data efficiency.

Original: This has led to the burgeoning field of differentially private optimizationalgorithms [5, 6, 20], which considers stochastic convex minimization as the canonical problem.
Revised: This has led to the burgeoning field of differentially private optimizationalgorithms [5, 6, 21], which considers stochastic convex minimization as the canonical problem.

Original: For differentially private stochastic convex optimization (DP-SCO) with ( ε, δ ) -DP guarantees, theoptimal excess population risk is Θ(1 / ( µn ) + d log(1 /δ ) /
Revised: For differentially private stochastic convex optimization (DP-SCO) with ( ε, δ ) -DP guarantees, theoptimal excess population risk is Θ(1 / ( µn ) + d log(1 /δ ) /

Original: ( µn 2 ε 2 )) for µ -strongly convex objectives,where n is the number of participants and d is the dimension of the variable.
Revised: ( µn 2 ε 2 )) for µ -strongly convex objectives,where n is the number of participants and d is the dimension of the variable.

Original: If the objective isalso smooth, this can be achieved with a linear-time ( ε, δ ) -DP algorithm
Revised: If the objective isalso smooth, this can be achieved with a linear-time ( ε, δ ) -DP algorithm

Original: [20] using O ( n ) stochasticgradient evaluations.
Revised: [21] using O ( n ) stochasticgradient evaluations.

Original: This analysis critically relies on the concept of algorithmic stability [11], whichmeasures how much the population loss of an algorithm’s output changes when a single data pointin the input dataset is perturbed.
Revised: This analysis critically relies on the concept of algorithmic stability [11], whichmeasures how much the population loss of an algorithm’s output changes when a single data pointin the input dataset is perturbed.

Original: This is also known as sensitivity of an algorithm in DP, whichdetermines how much noise needs to be added in order to achieve ( ε, δ ) -DP
Revised: This is also known as sensitivity of an algorithm in DP, whichdetermines how much noise needs to be added in order to achieve ( ε, δ ) -DP

Original: (see Definition 2).
Revised: (see Definition 2).

Original: Wewill use stability and sensitivity interchangeably when referring to optimization algorithms.
Revised: Wewill use stability and sensitivity interchangeably when referring to optimization algorithms.

Original: As thesensitivity of stochastic gradient descent (SGD) is known [22], Feldman et al.
Revised: As thesensitivity of stochastic gradient descent (SGD) is known [23], Feldman et al.

Original: [20] was able to addan appropriate amount of noise—tailored to this sensitivity—to an SGD-based algorithm to achieveoptimal risk in linear time.
Revised: [21] was able to addan appropriate amount of noise—tailored to this sensitivity—to an SGD-based algorithm to achieveoptimal risk in linear time.

Original: However, such tight analysis of sensitivity is generally intractable for morecomplex optimization routines that practitioners might want to use.
Revised: However, such tight analysis of sensitivity is generally intractable for morecomplex optimization routines that practitioners might want to use.

Original: Further, the stability analysis of
Revised: Further, the stability analysis of

Original: SGD only holds when the smoothness parameter ℓ of the problem is upper-bounded by ˜ O ( µn ) .
Revised: SGD only holds when the smoothness parameter ℓ of the problem is upper-bounded by ˜ O ( µn ) .

Original: Inthe first part of the paper, we show that we can achieve similar guarantees using a wider choice of,potentially more practical, “base” algorithms, without any restrictions on the smoothness parameter.
Revised: Inthe first part of the paper, we show that we can achieve similar guarantees using a wider choice of,potentially more practical, “base” algorithms, without any restrictions on the smoothness parameter.

Original: On another front, many emerging practical machine learning applications are formulated as stochasticminimax optimization problems, e.g., generative adversarial networks [21], adversarially robustmachine learning [32], and reinforcement learning [15].
Revised: On another front, many emerging practical machine learning applications are formulated as stochasticminimax optimization problems, e.g., generative adversarial networks [22], adversarially robustmachine learning [34], and reinforcement learning [15].

Original: Designing DP algorithms for solving theseminimax problems is of paramount importance.
Revised: Designing DP algorithms for solving theseminimax problems is of paramount importance.

Original: For example, private generative adversarial networksprovide a promising new direction to synthetic data generation [42], such as in networked time-seriesdata [30].
Revised: For example, private generative adversarial networksprovide a promising new direction to synthetic data generation [45], such as in networked time-seriesdata [31].

Original: Motivated by such applications, we study the differentially private stochastic minimaxoptimization (DP-SMO) problem of the form:
Revised: Motivated by such applications, we study the differentially private stochastic minimaxoptimization (DP-SMO) problem of the form:

Original: [Equation]
Revised: [Equation]

Original: where the objective f ( x, y ; ξ ) is smooth and convex-concave for any random vector ξ , and we aregiven access to n i.i.d.
Revised: where the objective f ( x, y ; ξ ) is smooth and convex-concave for any random vector ξ , and we aregiven access to n i.i.d.

Original: samples { ξ i } ni =1 .
Revised: samples { ξ i } ni =1 .

Original: In contrast to DP-SCO, where linear-time algorithms havebeen proposed to achieve optimal risk guarantees, existing private algorithms [10, 44] for DP-SMOachieving optimal guarantees have time-complexity which scales super-linearly in the number ofsamples n (summarized in Table 1).
Revised: In contrast to DP-SCO, where linear-time algorithms havebeen proposed to achieve optimal risk guarantees, existing private algorithms [10, 47] for DP-SMOachieving optimal guarantees have time-complexity which scales super-linearly in the number ofsamples n (summarized in Table 1).

Original: In the second part of this paper, we close this gap by introducinga new class of output perturbation-based DP algorithms for both strongly-convex–strongly-concaveand convex-concave settings, which can achieve near-optimal population risk bounds using ˜ O ( n )stochastic gradient computations, where ˜ O hides logarithmic factors.
Revised: In the second part of this paper, we close this gap by introducinga new class of output perturbation-based DP algorithms for both strongly-convex–strongly-concaveand convex-concave settings, which can achieve near-optimal population risk bounds using ˜ O ( n )stochastic gradient computations, where ˜ O hides logarithmic factors.

Original: One of the main bottlenecks for (i) using a wider variety of algorithms to solve the DP-SCO problem,and (ii) solving the DP-SMO problem using linear-time algorithms, is the lack of known stabilityresults for fast and sophisticated non-private algorithms, such as variance-reduced and acceleratedmethods.
Revised: One of the main bottlenecks for (i) using a wider variety of algorithms to solve the DP-SCO problem,and (ii) solving the DP-SMO problem using linear-time algorithms, is the lack of known stabilityresults for fast and sophisticated non-private algorithms, such as variance-reduced and acceleratedmethods.

Original: One can sidestep the algorithm-specific stability requirement by utilizing the stability ofthe optimal solution to a strongly-convex and Lipschitz empirical risk minimization (ERM) problem[39, 46].
Revised: One can sidestep the algorithm-specific stability requirement by utilizing the stability ofthe optimal solution to a strongly-convex and Lipschitz empirical risk minimization (ERM) problem[41, 49].

Original: Particularly, if the output of an algorithm for such an ERM problem is close enough to itsempirical solution, then the sensitivity of the algorithm is automatically guaranteed.
Revised: Particularly, if the output of an algorithm for such an ERM problem is close enough to itsempirical solution, then the sensitivity of the algorithm is automatically guaranteed.

Original: Exploiting thisobservation, Feldman et al.
Revised: Exploiting thisobservation, Feldman et al.

Original: [20] proposed the phased-ERM algorithm for nonsmooth DP-SCO, whichsolves a series of strongly-convex ERM subproblems to sidestep the stability analysis of SGD fornonsmooth functions.
Revised: [21] proposed the phased-ERM algorithm for nonsmooth DP-SCO, whichsolves a series of strongly-convex ERM subproblems to sidestep the stability analysis of SGD fornonsmooth functions.

Original: This algorithm achieves a quadratic time-complexity for nonsmooth DP-SCO.
Revised: This algorithm achieves a quadratic time-complexity for nonsmooth DP-SCO.

Original: We observe that when the problem is additionally smooth, or when it allows a smooth minimaxreformulation [34], there exist a plethora of fast algorithms to solve the resulting ERM subproblems.
Revised: We observe that when the problem is additionally smooth, or when it allows a smooth minimaxreformulation [36], there exist a plethora of fast algorithms to solve the resulting ERM subproblems.

Original: Combining these algorithms with (phased) output perturbation gives rise to a class of near-optimalnear-linear  time-complexity private algorithms for both smooth DP-SCO and DP-SMO problems,without any additional effort on their stability analysis.
Revised: Combining these algorithms with (phased) output perturbation gives rise to a class of near-optimalnear-linear 1 time-complexity private algorithms for both smooth DP-SCO and DP-SMO problems,without any additional effort on their stability analysis.

Original: Our contributions are summarized below:
Revised: Our contributions are summarized below:

Original: • We introduce a flexible framework for solving smooth DP-SMO and smooth DP-SCO problems,utilizing the (phased) output perturbation mechanism.
Revised: • We introduce a flexible framework for solving smooth DP-SMO and smooth DP-SCO problems,utilizing the (phased) output perturbation mechanism.

Original: The black-box framework enables us to bypassthe need to prove algorithm-specific stability and transform off-the-shelf optimization algorithmsinto DP algorithms with near-optimal guarantees.
Revised: The black-box framework enables us to bypassthe need to prove algorithm-specific stability and transform off-the-shelf optimization algorithms [Table] into DP algorithms with near-optimal guarantees.

Original: This is attractive as there are currently no stabilityanalyses for accelerated [2, 36] and variance-reduced [24, 36] algorithms for both SMO and SCO.
Revised: This is attractive as there are currently no stabilityanalyses for accelerated [2, 38] and variance-reduced [25, 38] algorithms for both SMO and SCO.•

Revised: Using the framework

Original: [Table]
Revised: [Table]

Original: • Using the framework, we provide the first linear-time private algorithms for smooth DP-SMOwith optimal bound on the population risk up to logarithmic factors, under both strongly-convex–strongly-concave and convex-concave cases (see Table 1).
Revised: we provide the first near-linear time private algorithms for smooth DP-SMOwith near-optimal bound on the population duality gap, under both strongly-convex–strongly-concave and convex-concave cases (see Table 1).

Original: Among other things, this implies that if a(primal) nonsmooth minimization problem can be reformulated as a smooth convex-concave minimaxoptimization, then we can solve it in near-linear time instead of the best known super-linear time[4, 28].
Revised: Among other things, this implies that if a (primal) nonsmoothminimization problem can be reformulated as a smooth convex-concave minimax optimization, thenwe can solve it in near-linear time instead of the best known super-linear time [4, 29].

Original: In prior work, linear-time DP algorithms with optimal rates for solving nonsmooth objectivesonly existed for generalized linear losses [8].• The framework also enriches the cohort of near-linear time optimal private algorithms for smoothDP-SCO settings, that only contained SGD previously.
Revised: In priorwork, linear-time DP algorithms with optimal rates for solving nonsmooth objectives only existed forgeneralized linear losses [8].• The framework also enriches the cohort of near-linear time near-optimal private algorithms forsmooth DP-SCO settings, that only contained SGD previously.

Original: Moreover, existing optimal DP algorithmsrely on a stability result of SGD, which holds only for a restricted range of the smoothness parameter[22].
Revised: Moreover, existing optimal DPalgorithms for smooth DP-SCO rely on a stability result of SGD, which holds only for a restrictedrange of the smoothness parameter [23].

Original: Such restrictions are avoided in our framework.
Revised: Such restrictions are avoided in our framework.

Original: Related Works Differentially private optimization has been an active research field for the pastfew years, and early works focused on the empirical problems [5, 45, 41].
Revised: Related Works Differentially private optimization has been an active research field for the past few years, and early works focused on the empirical problems [5, 48, 43].

Original: In addition to the outputperturbation used in this paper, many existing works applied gradient perturbation to guarantee privacy.
Revised: In addition to the output perturbation used in this paper, many existing works applied gradient perturbation to guarantee privacy.

Original: This method adds noise to each iteration of the algorithm and then applies moments accountant [1] oradvanced composition [25] to analyze the overall privacy.
Revised: This method adds noise to each iteration of the algorithm and then applies moments accountant [1] or advanced composition [26] to analyze the overall privacy.

Original: Although gradient perturbation does notneed the smoothness or convexity assumptions of the function and works for most iterative algorithms,it requires larger mini-batches [6] or longer training time [7] if one resorts to the privacy amplificationvia subsampling [9] to reduce the DP noise, resulting in super-linear gradient queries in existingmethods [6, 44].
Revised: Although gradient perturbation does notneed the smoothness or convexity assumptions of the function and works for most iterative algorithms, it requires larger mini-batches [6] or longer training time [7] if one resorts to the privacy amplificationvia subsampling [9] to reduce the DP noise, resulting in super-linear gradient queries in existingmethods [6, 47].

Original: Here we mainly review previous results that achieve the optimal utility bound
Revised: Here we mainly review previous results that achieve the optimal utility bound

Original: O (1 / √ n + (cid:112) d log(1 /δ ) / ( nε )) on the population loss—according to a lower-bound in Bassily et al.[5]—when solving a d dimensional DP-SCO problem using n samples with ( ε, δ ) -DP guarantees.
Revised: O (1 / √ n + (cid:112) d log(1 /δ ) / ( nε )) on the population loss—according to a lower-bound in Bassily et al.[5]—when solving a d dimensional DP-SCO problem using n samples with ( ε, δ ) -DP guarantees.

Original: In the smooth convex case, Bassily et al.
Revised: In the smooth convex case, Bassily et al.

Original: [6] is the first to derive optimal rates for DP-SCO withcomplexity O ( n 3 / 2 √ ε ) by gradient perturbation and stability of SGD [22].
Revised: [6] is the first to derive optimal rates for DP-SCO withcomplexity O ( n 3 / 2 √ ε ) by gradient perturbation and stability of SGD [23].

Original: Feldman et al.
Revised: Feldman et al.

Original: [20]provided two SGD-based linear-time algorithms: one uses privacy amplification by iteration [19] thatonly works for contractive updates; the second one uses phased output perturbation and stability of
Revised: [21]provided two SGD-based linear-time algorithms: one uses privacy amplification by iteration [20] thatonly works for contractive updates; the second one uses phased output perturbation and stability of

Original: SGD.
Revised: SGD.

Original: In the nonsmooth convex case, Bassily et al.
Revised: In the nonsmooth convex case, Bassily et al.

Original: [7] established the stability of SGD for nonsmoothfunctions and obtained a quadratic-time algorithm.
Revised: [7] established the stability of SGD for nonsmoothfunctions and obtained a quadratic-time algorithm.

Original: Feldman et al.
Revised: Feldman et al.

Original: [20] leveraged the stability of ERMsolutions [39] and achieved O ( n 2 log(1 /δ ))
Revised: [21] leveraged the stability of ERMsolutions [41] and achieved O ( n 2 log(1 /δ ))

Original: complexity with phased output perturbation.
Revised: complexity with phased output perturbation.

Original: Based onthis phased framework, Asi et al.
Revised: Based onthis phased framework, Asi et al.

Original: [4] and Kulkarni et al.
Revised: [4] and Kulkarni et al.

Original: [28] used gradient perturbation and improvedthe complexity to O (min { n 3 / 2 , n 2 / √ d } ) and O (min { n 5 / 4 d 1 / 8 , n 3 / 2 /d 1 / 8 } ) respectively; Asi et al.[3] introduced a hypothetical linear-time algorithm assuming the existence of a low-biased estimator.
Revised: [29] used gradient perturbation and improvedthe complexity to O (min { n 3 / 2 , n 2 / √ d } ) and O (min { n 5 / 4 d 1 / 8 , n 3 / 2 /d 1 / 8 } ) respectively; Asi et al.[3] introduced a hypothetical linear-time algorithm assuming the existence of a low-biased estimator.

Original: Moreover, Bassily et al.
Revised: Moreover, Bassily et al.

Original: [8] gave a linear-time algorithm for generalized linear losses using phased
Revised: [8] gave a linear-time algorithm for generalized linear losses using phased

Original: SGD [20] with smoothing techniques.
Revised: SGD [21] with smoothing techniques.

Original: To the best of our knowledge, only few papers studied DP-SMO and all of them used gradientperturbation to guarantee privacy.
Revised: To the best of our knowledge, only few papers studied DP-SMO and all of them used gradientperturbation to guarantee privacy.

Original: Boob and Guzmán [10] analyzed stability of Extragradient [40] andproximal point methods [38] for smooth convex-concave functions, and their DP versions run with time O ( n 2 ) and ˜ O ( n 3 / 2 ) respectively.
Revised: Boob and Guzmán [10] analyzed stability of Extragradient [42] and proximal point methods [40] for smooth convex-concave functions, and their DP versions run withtime O ( n 2 ) and ˜ O ( n 3 / 2 ) respectively.

Original: Yang et al.
Revised: Yang et al.

Original: [44] used the stability of SGDA [29] and obtained
Revised: [47] used the stability of SGDA [30] and obtained

Original: DP algorithms with complexity O ( n 3 / 2 √ ε ) for the smooth convex-concave case and O ( n 2 ) for thenonsmooth case, mirroring the guarantees of SGD for DP-SCO [6, 7].
Revised: DP algorithms with complexity O ( n 3 / 2 √ ε ) for the smooth convex-concave case and O ( n 2 ) for thenonsmooth case, mirroring the guarantees of SGD for DP-SCO [6, 7].

Original: Kang et al.
Revised: Kang et al.

Original: [27] only focusedon stability and generalization analysis of gradient perturbed SGDA and provided high-probabilityresults.
Revised: [28] only focusedon stability and generalization analysis of gradient perturbed SGDA and provided high-probabilityresults.

Original: See Table 1 for a brief comparison.
Revised: See Table 1 for a brief comparison.

Revised: The idea of using the stability of ERM for smooth DP-SCO was also exploited in Lowy and

Original: Notations We use ∥·∥ for the Euclidean norm of a vector and |·| for the absolute value or thecardinality of a set.
Revised: Razaviyayn [32]. However, they restricted to output perturbation and only provided near-linear timenear-optimal algorithms for smooth strongly-convex losses. For smooth convex case, they achieved asub-optimal rate O (1 / √ n + ( (cid:112) d log(1 /δ ) / ( nε )) 2 / 3 ) in near-linear time.

Revised: In the setting of smoothstrongly-convex–(strongly-)concave DP-SMO, they directly applied the DP-SCO reformulation ofthe primal function to obtain the final guarantees.

Revised: Therefore, their utility bound is simply on theprimal risk, which is weaker than the primal-dual gap considered in this work.

Revised: The algorithms run innear-linear time for the strongly-concave case and super-linear time O ( n 5 / 2 ) for the concave case.

Revised: Incontrast, we provide near-optimal near-linear time algorithms in all aforementioned settings.

Revised: Notations: We use ∥·∥ for the Euclidean norm of a vector and |·| for the absolute value or thecardinality of a set.

Original: A function g : R d → R is L -Lipschitz if | g ( x 1 ) − g ( x 2 ) | ≤ L ∥ x 1 − x 2 ∥for x 1 , x 2 in the domain of g .
Revised: A function g : R d → R is L -Lipschitz if | g ( x 1 ) − g ( x 2 ) | ≤ L ∥ x 1 − x 2 ∥for x 1 , x 2 in the domain of g .

Original: A function h : R d → R is ℓ -smooth if it is differentiable andh ( x 2 )
Revised: A function h : R d → R is ℓ -smooth if it is differentiable andh ( x 2 )

Original: ≤ h ( x 1 ) + ∇ h ( x 1 ) ⊤ ( x 2 − x 1 ) + ( ℓ/ 2) ∥ x 1 − x 2 ∥ 2 .
Revised: ≤ h ( x 1 ) + ∇ h ( x 1 ) ⊤ ( x 2 − x 1 ) + ( ℓ/ 2) ∥ x 1 − x 2 ∥ 2 .

Original: A function p : R d → R is convex ifp ( αx 1 + (1 − α ) x 2 ) ≤ αp ( x 1 ) + (1 − α ) p ( x 2 ) for all α ∈ [0 , 1] , and p is µ -strongly convex ifp ( x ) − ( µ/ 2)
Revised: A function p : R d → R is convex ifp ( αx 1 + (1 − α ) x 2 ) ≤ αp ( x 1 ) + (1 − α ) p ( x 2 ) for all α ∈ [0 , 1] , and p is µ -strongly convex ifp ( x ) − ( µ/ 2)

Original: ∥ x ∥ 2 is convex with µ > 0 .
Revised: ∥ x ∥ 2 is convex with µ > 0 .

Original: A function q : R d → R is concave if − q is convex andµ -strongly concave if − q is µ -strongly convex.
Revised: A function q : R d → R is concave if − q is convex andµ -strongly concave if − q is µ -strongly convex.

Original: For a vector x ∈ R d , the notation x + N (0 , σ 2 I d )means x + z for a random vector z ∼ N (0 , σ 2 I d ) sampled from the Gaussian distribution.
Revised: For a vector x ∈ R d , the notation x + N (0 , σ 2 I d )means x + z for a random vector z ∼ N (0 , σ 2 I d ) sampled from the Gaussian distribution.

Original:  Preliminaries
Revised:  Preliminaries

Original: We first provide some background on differential privacy and stochastic minimax optimization.
Revised: We first provide some background on differential privacy and stochastic minimax optimization.

Original: 2.1 Differential Privacy
Revised: 2.1 Differential Privacy

Original: Differential Privacy (DP), introduced in Dwork et al.
Revised: Differential Privacy (DP), introduced in Dwork et al.

Original: [17], measures privacy leakage of an algorithm.
Revised: [17], measures privacy leakage of an algorithm.

Original: Definition 1.
Revised: Definition 1.

Original: For two datasets S = { ξ i } ni =1 and S ′
Revised: For two datasets S = { ξ i } ni =1 and S ′

Original: = { ξ ′ i } ni =1 , we say the pair ( S, S ′ ) is neighboringif max {| S \ S ′
Revised: = { ξ ′ i } ni =1 , we say the pair ( S, S ′ ) is neighboringif max {| S \ S ′

Original: | , | S ′ \ S |} = 

Revised: | , | S ′ \ S |} =

Original: and we denote neighboring datasets with S ∼ S ′ .
Revised: and we denote neighboring datasets with S ∼ S ′ .

Original: For an algorithm Aand some privacy parameters ε > 0 and δ ∈ (0 , 1) , we say A satisfies ( ε, δ ) - differential privacy if 143 P ( A ( S ) ∈
Revised: For an algorithm Aand some privacy parameters ε > 0 and δ ∈ (0 , 1) , we say A satisfies ( ε, δ ) - differential privacy if 152 P ( A ( S ) ∈

Original: A )
Revised: A )

Original: ≤ e ε P
Revised: ≤ e ε P

Original: ( A ( S ′ )
Revised: ( A ( S ′ )

Original: ∈ A )
Revised: ∈ A )

Original: + δ for all S ∼ S ′ and all subset A in the range of A .
Revised: + δ for all S ∼ S ′ and all subset A in the range of A .

Revised: In this work, we focus on the settings when ε ∈ (0 , 1) and δ ∈ (0 , 1 /n ) given dataset of size n .

Original: Sensitivity is an important concept that makes designing ( ε, δ )
Revised: Sensitivity is an important concept that makes designing ( ε, δ )

Original: -DP mechanisms straightforward.
Revised: -DP mechanisms straightforward.

Original: Definition 2.
Revised: Definition 2.

Original: A randomized algorithm A operating on S and outputting a vector in R d has sensitivity∆ A := sup S ∼ S
Revised: A randomized algorithm A operating on S and outputting a vector in R d has sensitivity∆ A := sup S ∼ S

Original: ′ ∥A ( S )
Revised: ′ ∥A ( S )

Original: − A ( S ′ )
Revised: − A ( S ′ )

Original: ∥ with probability at least 1 − δ .
Revised: ∥ with probability at least 1 − δ .

Original: For this query, the Gaussianmechanism outputs A ( S ) + N (0 , (∆ A (cid:112) 2 log(1 .
Revised: For this algorithm, the Gaussianmechanism outputs A ( S ) + N (0 , (∆ A (cid:112) 2 log(1 .

Original: 25 /δ ) /ε ) 2 I d ) and achieves ( ε, 2 δ ) -DP
Revised: 25 /δ ) /ε ) 2 I d ) and achieves ( ε, 2 δ ) -DP

Original: The following basic composition rule of differential privacy will be used in the analysis.
Revised: The following basic composition rule of differential privacy will be used in the analysis.

Original: Lemma 2.1.
Revised: Lemma 2.1.

Original: If A 1 is ( ε 1 , δ 1 ) -DP
Revised: If A 1 is ( ε 1 , δ 1 ) -DP

Original: and A 2 is ( ε 2 , δ 2 )
Revised: and A 2 is ( ε 2 , δ 2 )

Original: -DP, then ( A 1 , A 2 ) is ( ε 1 + ε 2 , δ 1 + δ 2 ) -DP[16].
Revised: -DP, then ( A 1 , A 2 ) is ( ε 1 + ε 2 , δ 1 + δ 2 ) -DP[16].

Original: For a sequence of interactive queries {A k } Kk =1 each satisfying ( ε k , δ k ) -DP and operating on a subset S k , if S k ’s are disjoint then the composition ( A 1 ( S 1 ) , A 2 ( S 2 ) , . . .
Revised: For a sequence of interactive queries {A k } Kk =1 each satisfying ( ε k , δ k ) -DP and operating on a subset S k , if S k ’s are disjoint then the composition ( A 1 ( S 1 ) , A 2 ( S 2 ) , . . .

Original: , A K ( S K ))
Revised: , A K ( S K ))

Original: is(max k ∈ [ K ] ε k , max k ∈
Revised: is(max k ∈ [ K ] ε k , max k ∈

Original: [ K ] δ k ) -DP
Revised: [ K ] δ k ) -DP

Original: (known as parallel composition in McSherry [33]).
Revised: (known as parallel composition in McSherry [35]).

Original: In this work, we focus on the settings when ε ∈ (0 , 1) and δ ∈ (0 , 1 /n ) given dataset of size n

Original: 2.2 Stochastic Minimax Optimization
Revised: 2.2 Stochastic Minimax Optimization

Original: The stochastic minimax (a.k.a.
Revised: The stochastic minimax (a.k.a.

Original: saddle point) optimization problem has the form:
Revised: saddle point) optimization problem has the form:

Original: [Equation]
Revised: [Equation]

Original: where F is the population-level expectation of the stochastic continuous objective f ( · , · ; ξ ) :
Revised: where F is the population-level expectation of the stochastic continuous objective f ( · , · ; ξ ) :

Original: R d x ×
Revised: R d x ×

Original: R d y → R with closed convex domains X ⊂ R d x and Y ⊂ R d y whose stochasticity is captured bythe random vector ξ .
Revised: R d y → R with closed convex domains X ⊂ R d x and Y ⊂ R d y whose stochasticity is captured by

Revised: Input: Dataset S = { ξ i } ni =1 , algorithm A , DP parameters ( ε, δ ) , strongly-convex parameter µ .

Revised: x ξ i obtain the output A ( S ) such that ∥A ( S ) − ˆ x ∗ S ∥ ≤ L/ ( µn ) with probability at least 1 − δ/ 4 ,

Revised: ˆ x ∗ S = arg min x ∈X ˆ F S ( x ) is the empirical optimal solution.

Revised: Output: d log δ ) /

Revised: the random vector ξ .

Original: We are interested in the population saddle point ( x ∗ , y ∗ ) ∈
Revised: We are interested in the population saddle point ( x ∗ , y ∗ ) ∈

Original: X × Y of the aboveproblem such that F ( x ∗ , y ) ≤ F ( x ∗ , y ∗ ) ≤ F ( x, y ∗ ) for all ( x, y ) ∈
Revised: X × Y of the aboveproblem such that F ( x ∗ , y ) ≤ F ( x ∗ , y ∗ ) ≤ F ( x, y ∗ ) for all ( x, y ) ∈

Original: X × Y .
Revised: X × Y .

Original: For some randomized algorithm with output (˜ x, ˜ y ) , we measure its convergence by the population strong duality gap E [max y ∈Y F (˜ x, y )
Revised: For some randomized algorithm with output (˜ x, ˜ y ) , we measure its convergence by the population strong duality gap E [max y ∈Y F (˜ x, y )

Original: − min x ∈X F ( x, ˜ y )] or the population weak duality gapmax y ∈Y E
Revised: − min x ∈X F ( x, ˜ y )] or the population weak duality gapmax y ∈Y E

Original: [ F (˜ x, y )] − min x ∈X E [ F ( x, ˜ y )] .
Revised: [ F (˜ x, y )] − min x ∈X E [ F ( x, ˜ y )] .

Original: Note that both duality gaps are always larger than 0 , anda deterministic
Revised: Note that both duality gaps are always larger than 0 , anda deterministic

Original: (˜ x, ˜ y ) is the saddle point if and only if the gaps are 0 .
Revised: (˜ x, ˜ y ) is the saddle point if and only if the gaps are 0 .

Original: In practice, we usually do not have access to the distribution P ξ of the random vector ξ .
Revised: In practice, we usually do not have access to the distribution P ξ of the random vector ξ .

Original: Instead weare given a dataset S = { ξ i } ni =1 with n random vectors independently sampled from the distribution
Revised: Instead weare given a dataset S = { ξ i } ni =1 with n random vectors independently sampled from the distribution

Original: Input: Dataset S = { ξi } ni =1 , algorithm A , DP parameters ( ε, δ ) .
Revised: P ξ .

Original: 1: Run the algorithm A on the smooth strongly-convex finite-sum problem min x ∈X ˆ F S ( x ) =
Revised: We define the empirical minimax optimization problem as

Original: (1 /n )
Revised: [Equation]

Original: (cid:80) ni
Revised: (cid:88)

Original: =1 f ( x ; ξ i ) to obtain the output A ( S ) such that ∥A ( S ) − ˆ x ∗ S ∥ ≤ L/ ( µn ) with probability at least 1 − δ/ 4 , where ˆ x ∗ S = arg min x ∈X ˆ F S ( x ) is the empirical optimal solution. Output: ˜ x
Revised: [Equation]

Original: =A ( S ) + N (0 , σ 2 I d ) with σ = 4 L (cid:112) 2 log(2 . 5 /δ ) / ( µnε ) . P ξ . We define the empirical minimax optimization problem as
Revised: =

Original: [Equation]
Revised: [Equation]

Revised: f ( x

Original:  Similarly, we can define the empirical saddle point (ˆ x ∗ S , ˆ y ∗ S ) and the empirical duality gap w.r.t.
Revised: [Equation] y [Equation] Similarly, we can define the empirical saddle point (ˆ x ∗ S , ˆ y ∗ S ) and the empirical duality gap w.r.t.

Original: theempirical function ˆ F S .
Revised: theempirical function ˆ F S .

Original: Zhang et al.
Revised: Zhang et al.

Original: [46] established the stability and generalization properties of theempirical saddle point, which is essential to the design of our DP-SMO algorithms.
Revised: [49] established the stability and generalization properties of theempirical saddle point, which is essential to the design of our DP-SMO algorithms.

Original:  Differentially Private Stochastic Convex Optimization
Revised:  Differentially Private Stochastic Convex Optimization

Original: As a warm-up, we start with a simpler problem of differentially private stochastic convex optimization(DP-SCO) to showcase our main ideas.
Revised: As a warm-up, we start with a simpler problem of differentially private stochastic convex optimization(DP-SCO) to showcase our main ideas.

Original: We consider the following stochastic optimization:
Revised: We consider the following stochastic optimization:

Original: [Equation]
Revised: [Equation]

Original: where the stochastic function f ( · , ξ ) : R d → R is defined on a convex domain X ⊂ R d and ξ is arandom vector from an unknown distribution P ξ .
Revised: where the stochastic function f ( · , ξ ) : R d → R is defined on a convex domain X ⊂ R d and ξ is arandom vector from an unknown distribution P ξ .

Original: Given a dataset S with n i.i.d.
Revised: Given a dataset S with n i.i.d.

Original: samples from P ξ , wedevelop a generic output perturbation framework with both privacy and population loss guarantees.
Revised: samples from P ξ , wedevelop a generic output perturbation framework with both privacy and population loss guarantees.

Original: 3.1 Near-Linear Time Algorithms for  Strongly-Convex Functions
Revised: 3.1 Near-Linear Time Algorithms for Smooth Strongly-Convex Functions

Original: First we study the case when the objective function is strongly-convex with the following assumptions.
Revised: First we study the case when the objective function is strongly-convex with the following assumptions.

Original: Assumption 3.1.
Revised: Assumption 3.1.

Original: For any ξ , the function f ( x ; ξ ) is L -Lipschitz, ℓ -smooth, and convex on the closedconvex domain X ⊂ R d .
Revised: For any ξ , the function f ( x ; ξ ) is L -Lipschitz, ℓ -smooth, and convex on the closedconvex domain X ⊂ R d .

Original: Assumption 3.2.
Revised: Assumption 3.2.

Original: For any ξ , f ( x ; ξ ) satisfies Assumption 3.1 and it is µ -strongly convex on X .
Revised: For any ξ , f ( x ; ξ ) satisfies Assumption 3.1 and it is µ -strongly convex on X .

Original: Shalev-Shwartz et al.
Revised: Shalev-Shwartz et al.

Original: [39] proved that the empirical optimal solution has bounded stability if theobjective f ( · ; ξ ) is strongly-convex and Lipschitz w.r.t.
Revised: [41] proved that the empirical optimal solution has bounded stability if theobjective f ( · ; ξ ) is strongly-convex and Lipschitz w.r.t.

Original: its domain (see Lemma A.2 in the appendix).
Revised: its domain (see Lemma A.2 in the appendix).

Original: As long as the output of some algorithm A is close enough to the empirical solution, we can showthat A has bounded sensitivity, and thus standard Gaussian mechanism (Definition 2) can be appliedto ensure differential privacy.
Revised: As long as the output of some algorithm A is close enough to the empirical solution, we can showthat A has bounded sensitivity, and thus standard Gaussian mechanism (Definition 2) can be appliedto ensure differential privacy.

Original: We formalize this in Algorithm 1, where A is any algorithm for solvingsmooth strongly-convex finite-sum minimization problems.
Revised: This is formalized in Algorithm 1, where A is any algorithm for solvingsmooth strongly-convex finite-sum minimization problems.

Original: The theorem below shows that Algorithm1 is ( ε, δ ) -DP with near-optimal guarantees on the excess risk.
Revised: The theorem below shows that Algorithm1 is ( ε, δ ) -DP with near-optimal guarantees on the excess risk.

Original: A proof is provided in Appendix A.
Revised: A proof is provided in Appendix A.

Original: Theorem 3.3.
Revised: Theorem 3.3.

Original: Under Assumption 3.2, Algorithm 1 is ( ε, δ ) -DP
Revised: Under Assumption 3.2, Algorithm 1 is ( ε, δ ) -DP

Original: and its output ˜ x satisfies(excess empirical risk)
Revised: and its output ˜ x satisfies(excess empirical risk)

Original: [Equation]
Revised: [Equation]

Original: population risk)
Revised: where κ = ℓ/µ is the condition number and we assume x ∗ and ˆ x ∗ S are interior points of X . min ∈X ˆ F k ( x ) ≜ (1 / ¯ n ) (cid:80) k ¯ n i =( k − 1)¯ n +1 f ( x ; ξ i )

Original: [Equation]
Revised: + ( µ k / 2) ∥ x − ˜ x

Original: where κ = ℓ/µ is the condition number and we assume x ∗ and ˆ x ∗ S are interior points of X .
Revised: k − 1 ∥ 2 to obtain the output x k such that ∥ x k − ˆ x ∗ k ∥ ≤ L/ ( µ k ¯ n ) with probability at least 1 − δ/ 4 , where ˆ x ∗ k = arg min x ∈X ˆ F k ( is the empirical optimal solution

Original: Remark 1.
Revised: Remark 1.

Original: The gradient complexity of Algorithm 1 is the complexity of A to find a solution suchthat ∥A ( S ) − ˆ x ∗ S

Original: ∥ ≤ L/ ( µn ) with probability at least 1 − δ/ 4 .

Original: Many algorithms for smooth stronglyconvex finite-sum minimization problems output solutions A ( S ) such that E
Revised: Many algorithms for smooth strongly-convex finite-sum minimization problems guarantee

Original:  [ ˆ F S ( A ( S )) − ˆ F S (ˆ x ∗ S )] ≤γ with O ( T ( n, κ ) log(1 /γ )) gradient evaluations, where T ( n, κ ) depends on the sample size nand the condition number κ .
Revised: E [ ˆ F S ( A ( S )) − ˆ F S (ˆ x ∗ S )] ≤ γ with O ( T ( n, κ ) log(1 /γ )) gradient evaluations, where n is the samplesize and κ is the condition number.

Original: Thus setting 1 /γ =
Revised: Setting 1 /γ =

Original: 32 µn 2 / ( δ 2 L 2 ) satisfies the requirements of Aby Markov’s inequality (see Appendix A).
Revised: 32 µn 2 / ( δ 2 L 2 ) satisfies the requirements of A by Markov’s inequality (see Appendix A).

Original: The gradient complexity of Algorithm 1 is O (( n +κ ) log( n/δ )) if A is SVRG [24] or SARAH [35], and O (( n + √ nκ ) log( n/δ )) if A is Katyusha [2].n (cid:80)
Revised: The gradient complexity of Algorithm 1 is O (( n +κ ) log( n/δ )) for SVRG [25] or SARAH [37], and O (( n + √ nκ ) log( n/δ )) for Katyusha [2].

Original: ( k − 1) n +1 f ( x ; ξ i )

Original: + ( µ k / 2)

Original: ∥ x −

Original: ˜ x

Original: k −

Original: ∥

Original: to

Original: obtain

Original: the output x k such that ∥ x k − ˆ x ∗ k ∥ ≤ L/ ( µ k ¯ n ) with probability at least 1 − δ/

Original: where ˆ x ∗ k =

Original: min ∈X is the empirical optimal solution As a comparison, most state-of-the-art private algorithms [41, 45, 13] for smooth strongly-convexfunctions only focus on empirical problems.
Revised: As a comparison, most state-of-the-art private algorithms [43, 48, 13] for smooth strongly-convexfunctions only focus on empirical problems.

Original: The linear-time algorithms in Feldman et al.
Revised: The linear-time algorithms in Feldman et al.

Original: [20] achieveoptimal population guarantees but only work for κ ≤ ˜ O ( n ) since they utilize the stability of SGD.
Revised: [21] achieveoptimal population guarantees but only work for κ ≤ ˜ O ( n ) since they utilize the stability of SGD.

Original: Instead, we provide a flexible framework that includes various base methods without the necessityto show their algorithm-specific stability.
Revised: Instead, we provide a flexible framework that includes various base methods without the necessity toshow their algorithm-specific stability.

Original: To the best of our knowledge, this is the first time thatsophisticated optimization algorithms besides SGD are proven to obtain near-optimal populationguarantees, without additional requirements on the problem’s parameters.

Original: Further, near-linear timecomplexity can be achieved when using fast variance reduction-based methods.
Revised: Further, near-linear time-complexity can be attained using fastvariance reduction-based methods.

Revised: Similar results existed in [32], but their base algorithm

Revised: A onlyhas in-expectation guarantees, which brings a critical challenge to the design of private mechanisms.

Original: 3.2 Near-Linear Time Algorithms for  Convex Functions
Revised: 3.2 Near-Linear Time Algorithms for Smooth Convex Functions

Original: Next, we study the convex setting.
Revised: Next, we study the convex setting.

Original: Here a direct method is to first reduce the convexproblemto a strongly-convex problem  by adding a regularizer ( µ/ 2)
Revised: A direct method is to first reduce the convex problem to a stronglyconvex one by adding a regularizer ( µ/ 2)

Original: ∥ x ∥ 2 to the objective and then apply Algorithm 1.
Revised: ∥ x ∥ 2 to the objective and then apply Algorithm 1.

Original: However, this approach only achieves a sub-optimal rate by a similar analysis as Zhanget al.
Revised: However,this approach only achieves a sub-optimal rate [48, 32].

Original: Inspired by the phased-ERM [20] method for nonsmooth convex objectives, we showthat a more sophisticated multi-phase Algorithm 2 using an increasing sequence of regularizationparameters { µ k } can achieve the near-optimal guarantees on the population loss for smooth DP-SCO.
Revised: Inspired by the phased-ERM [21] methodfor nonsmooth convex losses, we show that a more sophisticated multi-phase Algorithm 2 using anincreasing sequence of regularization parameters { µ k } can achieve the near-optimal guarantees onthe population loss for smooth DP-SCO.

Original: The increasing regularization parameter µ k in our algorithm ensures that both added DP noise andapproximation errors from the regularizer can be properly controlled.
Revised: The increasing regularization parameter µ k in the algorithmensures that both added DP noise and approximation errors from the regularizer can be properlycontrolled.

Original: The output of Algorithm 2satisfies the following guarantees, as proved in Appendix A.
Revised: The output of Algorithm 2 satisfies the following guarantees, as proved in Appendix A.

Original: Theorem 3.4.
Revised: Theorem 3.4.

Original: Let Assumption 3.1 hold.
Revised: Let Assumption 3.1 hold.

Original: Suppose the optimal solution x ∗ ∈ arg min x ∈X F ( x )satisfies that ∥ x ∗ ∥ ≤ D .
Revised: Suppose the optimal solution x ∗ ∈ arg min x ∈X F ( x )satisfies that ∥ x ∗ ∥ ≤ D .

Original: Then Algorithm 2 is ( ε, δ ) -DP and its output ˜ x K satisfies
Revised: Then Algorithm 2 is ( ε, δ ) -DP and its output ˜ x K satisfies

Original: [Equation]
Revised: [Equation]

Original: nε
Revised: for the excess population risk when setting µ

Original: [Equation]
Revised: [Equation]

Original: for the excess population risk when setting µ = ( L/D ) max (cid:8) 1 / √ n, 14 log( n ) (cid:112) d log(2 . 5 /δ ) / ( nε ) (cid:9)
Revised: L/ [Equation]

Original: [Equation]
Revised: [Equation]

Original: Remark 2.
Revised: Remark 2.

Original: The gradient complexity of Algorithm 2 is the total complexity of A to solve the smoothstrongly-convex finite-sum problem at each phase. By Remark 1 and the fact that (cid:80) Kk =1 1 /µ k ≤ O ( √ n ) , the  gradient complexity of Algorithm 2 is O (( n + √ nℓ ) log( n/δ )) if A is SVRG [24] or 
Revised: By Remark 1 and the fact that (cid:80) Kk =1 1 /µ k ≤ O (( D/L ) √ n ) , the total gradient complexity of Algorithm 2 is O (( n + √ nℓD/L ) log( n/δ )) using SVRG [25] or SARAH [37], and

Original: SARAH [35], and O (( n + n 3 / 4 √ ℓ ) log( n/δ ))
Revised: O (( n + n 3 / 4 (cid:112) ℓD/L ) log( n/δ ))

Original: if A is Katyusha [2] (seedetails in Appendix A).
Revised: for Katyusha [2] (see Appendix A).

Revised: For the sake of simplicity, thedependence on D/L in the complexity will be left out in the following.

Original: Wealso note that the smoothness assumption here is not necessary to derive the utility bound, but allowsthe use of fast accelerated algorithms for solving the regularized ERM problems.
Revised: We also point out that thesmoothness assumption here is not necessary to derive the utility bound, but allows the use of fastaccelerated algorithms for solving the regularized ERM problems.

Original: Bassily et al.
Revised: Bassily et al.

Original: [6] first proved optimal population guarantees for smooth convex functions, but theiralgorithm needs O ( n 3 / 2 √ ε ) gradient queries.
Revised: [6] first proved optimal population guarantees for smooth convex functions, but theiralgorithm needs O ( n 3 / 2 √ ε ) gradient queries.

Original: The algorithms in Feldman et al.
Revised: The algorithms in Feldman et al.

Original: [20] achieve lineartime-complexity, but require that ℓ ≤ O (max { √ n, (cid:112)
Revised: [21] achieve lineartime-complexity, and require that ℓ ≤ O (max {√ n, (cid:112)

Original: d log(1 /δ ) /ε } ) rooted in the stability analysisof SGD.
Revised: d log(1 /δ ) /ε } ) rooted in the stability analysisof SGD.

Original: In contrast, our framework can achieve near-optimal population guarantees using anyalgorithm without additional restrictions to the smoothness parameter.
Revised: In contrast, our framework can achieve near-optimal population guarantees using anyalgorithm without additional restrictions to the smoothness parameter.

Original: Note that the logarithmicfactors in the population risk bound can be removed with a different parameter selection (see Appendix
Revised: By equipping with variance

Original: A).
Revised: Input

Original: By equipping with variance reduction-based methods, we enrich the family of near-linear time
Revised: 1: Run algorithm A on smooth strongly-convex–strongly-concave finite-sum saddle point problem min y

Original: DP-SCO algorithms with optimal rates on the population loss.
Revised: ˆ

Original:  ) to obtain the output ( A x ( S ) , A y ( S )) such that with probability at least 1 − δ
Revised: S x ) = n ) 1 f ( x, y ξ i ) to obtain the output ( A x ( S ) , A y ( S )) such that with probability at least 1 − δ/ 8 ,

Original: [Equation]
Revised: [Equation]

Original:  where (ˆ x ∗ S , ˆ y ∗ S ) is the saddle point of ˆ F S ( x, y ) and we let µ :
Revised: (3) where (ˆ x ∗ S , ˆ y ∗ S ) is the saddle point of ˆ F S ( x, y ) and we let µ :

Original: = min { µ x , µ y } .
Revised: = min { µ x , µ y } .

Original: Differentially Private Stochastic Minimax Optimization
Revised: reduction-based methods, near-linear time-complexity can also be attained when ≤ O

Original: Using the same ideas as the minimization case in the previous section, we develop differentiallyprivate algorithms for stochastic minimax optimization:
Revised: √ the best of our knowledge, this is the first time that sophisticated optimization algorithms besides SGD are proven to obtain near-optimal population guarantees in near-linear time.

Original: [Equation]
Revised: Additionally, in theregime that Ω( √ n ) ≤ ℓ ≤ O ( n ) where previous smooth DP-SCO algorithms [6, 21] fail to provideoptimal guarantees, our framework still achieves a near-optimal rate with a better gradient complexity O ( n 3 / 4 √ ℓ ) using Katyusha compared to the best nonsmooth DP-SCO algorithms [4, 29].  Differentially Private Stochastic Minimax Optimization Using the same ideas as the minimization case in the previous section, we develop differentiallyprivate algorithms for stochastic minimax optimization in (1).

Original: 4.1 Near-Linear Time Algorithms for  Strongly-Convex–Strongly-Concave Functions
Revised: 4.1 Near-Linear Time Algorithms for Smooth Strongly-Convex–Strongly-Concave Functions

Original: First, we study the strongly-convex–strongly-concave (SC-SC) case with the following assumptions.
Revised: First, we study the strongly-convex–strongly-concave (SC-SC) case with the following assumptions.

Original: Assumption 4.1.
Revised: Assumption 4.1.

Original: For any vector ξ , f ( x, y ; ξ ) is L -Lipschitz and ℓ -smooth on the closed convexdomain X × Y ⊂ R d x × R d y .
Revised: For any vector ξ , f ( x, y ; ξ ) is L -Lipschitz and ℓ -smooth on the closed convexdomain X × Y ⊂ R d x × R d y .

Original: Moreover, f ( · , y ; ξ ) is convex on X for any y ∈ Y , and f ( x, · ; ξ ) isconcave on Y for any x ∈ X .
Revised: Moreover, f ( · , y ; ξ ) is convex on X for any y ∈ Y , and f ( x, · ; ξ ) isconcave on Y for any x ∈ X .

Original: Assumption 4.2.
Revised: Assumption 4.2.

Original: For any vector ξ , f ( x, y ; ξ ) satisfies Assumption 4.1 and f ( · , y ; ξ ) is µ x -stronglyconvex on X for any y ∈ Y , and f ( x, · ; ξ ) is µ y -strongly concave on Y for any x ∈ X .
Revised: For any vector ξ , f ( x, y ; ξ ) satisfies Assumption 4.1 and f ( · , y ; ξ ) is µ x -stronglyconvex on X for any y ∈ Y , and f ( x, · ; ξ ) is µ y -strongly concave on Y for any x ∈ X .

Original: Analogous to the minimization case, Zhang et al.
Revised: In a manner similar to the minimization case, Zhang et al.

Original: [46] showed that the optimal solution to theempirical saddle point problem is stable, which also implies its generalization error (see Lemma B.2in the appendix).
Revised: [49] showed that the empirical saddle pointis stable, which also implies its generalization error (see Lemma B.2 in the appendix).

Original: As a direct consequence, any algorithm A whose output is  closeenough to theempirical saddle point has bounded sensitivity.
Revised: As a directconsequence, any algorithm A whose output is sufficiently close to the empirical saddle point hasbounded sensitivity.

Original: This observation leads to Algorithm 3 for SC-SC DP-SMO with guarantees provided in the theorem below.
Revised: This observation leads to Algorithm 3 for SC-SC DP-SMO with guaranteesgiven in the theorem below.

Original: Here A can be any method for smooth SC-SC finite-sum minimax problems, and smoothness allows us to obtain efficient algorithms.
Revised: Here A can be any method for smooth SC-SC finite-sum minimaxproblems, and smoothness allows us to obtain efficient algorithms.

Original: Theorem 4.3.
Revised: Theorem 4.3.

Original: Under Assumption 4.2, let saddle points (ˆ x ∗ S , ˆ y ∗ S ) and ( x ∗ , y ∗ ) be interior points ofthe domain X × Y .
Revised: Under Assumption 4.2, let saddle points (ˆ x ∗ S , ˆ y ∗ S ) and ( x ∗ , y ∗ ) be interior points ofthe domain X × Y .

Original: Then Algorithm 3 is ( ε, δ ) -DP
Revised: Then Algorithm 3 is ( ε, δ ) -DP

Original: and its output (˜ x, ˜ y ) satisfies the following utilitybounds on the empirical and population strong duality gap:
Revised: and its output (˜ x, ˜ y ) satisfies the following utilitybounds on the empirical and population strong duality gap:

Original: [Equation]
Revised: [Equation]

Original:  where we let µ = min { µ x , µ y } , κ x = ℓ/µ x , κ y = ℓ/µ y , κ = ℓ/µ , and d = max { d x , d y } .
Revised: , where we let µ = min { µ x , µ y } , κ x = ℓ/µ x , κ y = ℓ/µ y , κ = ℓ/µ , and d = max { d x , d y } .

Original: Remark 3.
Revised: Remark 3.

Original: The gradient complexity of Algorithm 3 is the complexity of A to find a solutionsuch that (3) holds with probability at least 1 − δ/ 8 .

Original: Existing methods for finite-sum saddle pointproblems output solutions such that E [max y ∈Y ˆ F S
Revised: Existing methods for finite-sum saddle point problems output solutions such that E [Equation] max

Original: ( A x ( S ) , y )
Revised: [Equation]

Original: − min x ∈X ˆ F S ( x, A y ( S ))] ≤ γ withcomplexity O ( T ( n, κ ) log(1 /γ )) .
Revised: min [Equation] ∈X ˆ F S ( x, A y ( S ))] ≤ γ with complexity O ( T ( n, κ ) log(1 /γ )) .

Original: Thus setting 1 /γ = 

Revised: Thus setting 1 /γ =

Original: µn 2 / ( δL 2 ) satisfies the requirements of A(see Appendix B).
Revised: µn 2 / ( δL 2 ) satisfies the requirements of A (see Appendix B).

Original: The gradient complexity of Algorithm 3 is O ( nκ log( n/δ )) for Extragradient [40],
Revised: The gradientcomplexity of Algorithm 3 is O ( nκ log( n/δ )) for Extragradient [42], O (( n + κ 2 ) log( n/δ )) for

Original: O (( n + κ 2 ) log( n/δ )) for SVRG/SAGA [36], O (( n + √ nκ ) log( n/δ )) for Acc-SVRG/SAGA [36]and O (( n + √ nκ x κ y + n 3 / 4 √ κ ) log( n/δ )) for AL-SVRE [31] or Catalyst-Acc-SVRG [43].
Revised: SVRG/SAGA [38], O (( n + √ nκ ) log( n/δ )) for Acc-SVRG/SAGA [38] and O (( n + √ nκ x κ y +n 3 / 4 √ κ ) log( n/δ )) for AL-SVRE [33] or Catalyst-Acc-SVRG [46].

Original: The flexible framework allows the use of off-the-shelf optimization algorithms for smooth SC-SCfinite-sum minimax problems from a well-studied research community [43, 31] without knowing the

Original: [Equation]
Revised: [Equation]

Original: to obtain the output ( x k , y k ) such that with probability 1 − δ/ 8 ,
Revised: to obtain the output ( x k , y k ) such that with probability 1 − δ/ 8 ,

Original: [Equation]
Revised: [Equation]

Original: where (ˆ x ∗ k , ˆ y ∗ k ) is the saddle point of the regularized empirical function ˆ F k ( x, y ) .
Revised: where (ˆ x ∗ k , ˆ y ∗ k ) is the saddle point of the regularized empirical function ˆ F k ( x, y ) .

Original: algorithm-specific stability bound.
Revised: The minimax problem is equivalent to a minimization problem on x when the domain Y is restrictedto a singleton. As a result, the lower-bound Ω(1 / ( µ x n )+ d x log(1 /δ ) / ( µ x n 2 ε 2 )) of SC DP-SCO [5]trivially holds for SC-SC DP-SMO, and our results are near-optimal w.r.t. n and ( ε, δ ) . The flexibleframework allows the use of off-the-shelf optimization algorithms for smooth SC-SC finite-summinimax problems from a well-studied research community [46, 33] without being aware of thealgorithm-specific stability bound.

Original: Based on this framework, we can obtain the first near-linear timealgorithms for smooth SC-SC DP-SMO with near-optimal guarantees.
Revised: Based on this framework, we can produce the first near-linear timealgorithms for smooth SC-SC DP-SMO with near-optimal guarantees on the duality gap.

Revised: Similarrates were obtained in [32] on the primal risk through the reduction to SC DP-SCO.

Revised: However, thegeneralization error on the primal function does not always apply to the original minimax problemwhen the expectation and maximization cannot be exchanged.

Original: 4.2 Near-Linear Time Algorithms for  Convex-Concave Functions
Revised: 4.2 Near-Linear Time Algorithms for Smooth Convex-Concave Functions

Original: In this section, we consider the more general case where the objective is smooth and convex-concave.
Revised: In this section, we focus on the more general case when the objective is smooth and convex-concave.

Original: We still use the stability and generalization results of empirical solutions to construct our privatealgorithms.
Revised: We continue to build private algorithms using the stability and generalization results of empiricalsolutions.

Original: When the function is convex-concave,we can add a strongly-convex–strongly-concaveregularizer to the objective, and then apply the following results for the regularized empirical problems.
Revised: When the function is convex-concave, the following results for the regularized empiricalproblems can be applied by adding an SC-SC regularizer.

Original: Here we use a slightly tighter stability bound than the one in Zhang et al. [46] andprovide a proof  in Appendix B for completeness.
Revised: Here, the stability bound is slightly tighterthan the one in [49] and a proof is provided in Appendix B for completeness.

Original: Lemma 4.4.
Revised: Lemma 4.4.

Original: Consider a stochastic minimax problem such that f ( x, y ; ξ ) is convex-concave and LLipschitz with a µ x -strongly convex µ y -strongly concave regularizer G ( x, y ) .
Revised: Consider a stochastic minimax problem such that f ( x, y ; ξ ) is convex-concave and LLipschitz with a µ x -strongly convex µ y -strongly concave regularizer G ( x, y ) .

Original: Let µ = min { µ x , µ y }and denote the empirical saddle point of function ˆ F S ( x, y )
Revised: Let µ = min { µ x , µ y }and denote the empirical saddle point of function ˆ F S ( x, y )

Original: + G ( x, y ) as (ˆ x ∗ S , ˆ y ∗ S ) given dataset Swith n i.i.d.
Revised: + G ( x, y ) as (ˆ x ∗ S , ˆ y ∗ S ) given dataset Swith n i.i.d.

Original: samples.
Revised: samples.

Original: Then for any neighboring datasets S ∼ S ′ , we have
Revised: Then for any neighboring datasets S ∼ S ′ , we have

Original: [Equation]
Revised: [Equation]

Original: The stability result implies the generalization error of the empirical solution can be bounded as
Revised: The stability result implies the generalization error of the empirical solution can be bounded as

Original: [Equation]
Revised: [Equation]

Original: measured by the population weak duality gap.
Revised: measured by the population weak duality gap.

Original: In order to derive an optimal algorithm for convex-concave objectives, we need to carefully designthe regularization parameters for the primal variable x and the dual variable y to control bothapproximation errors and DP noise.
Revised: In order to derive a near-optimal algorithm for convex-concave objectives, we need to carefullydesign the regularization parameters for the primal variable x and the dual variable y to control bothapproximation errors and DP noise.

Original: Unlike DP-SCO, the interaction between the primal x and thedual y puts additional challenges in the analysis.
Revised: Unlike DP-SCO, the interaction between the primal x and thedual y puts additional challenges in the analysis.

Original: Based on the phased output perturbation framework,we propose a novel algorithm for convex-concave DP-SMO in Algorithm 4.
Revised: Based on the phased output perturbation framework,we propose a novel algorithm for convex-concave DP-SMO in Algorithm 4.

Original: Here, the fixed parameter µ ensures the approximation error from adding regularization for the dual isproperly bounded.
Revised: The fixed parameter µ ensures the approximation error from adding regularization for the dual isproperly bounded.

Original: The role of the increasing parameter µ k is the same as Algorithm 2.
Revised: The role of the increasing parameter µ k is the same as Algorithm 2.

Original: We only addnoise to the primal solutions and output ˜ x K in Algorithm 4.
Revised: We only add noise to the primal solutions and output ˜ x K in Algorithm 4.

Original: A natural way to derive a correspondingdual solution is to solve the smooth convex maximization problem max y ∈Y E [ F (˜ x K , y )] with DPconstraints.
Revised: A natural way to derive a correspondingdual solution is to solve the smooth concave maximization problem max y ∈Y E [ F (˜ x K , y )] with DPconstraints.

Original: We instead provide an alternative method that is symmetric to Algorithm 4 with the sameframework by switching the role of the primal and the dual.
Revised: We instead provide an alternative method that is symmetric to Algorithm 4 with the sameframework by switching the role of the primal and the dual.

Original: This phased algorithm iteratively solves
Revised: This phased algorithm iteratively solves

Revised: [Equation]

Revised: 

Original: [Equation]
Revised: [Equation]

Original: at phase k and only perturbs the dual variables.
Revised: at phase k and only perturbs the dual variables.

Original: The detailed Algorithm 5 is provided in Appendix B.
Revised: The detailed Algorithm 5 is provided in Appendix B.

Original: Its output ˜ y K together with the output ˜ x K of Algorithm 4 satisfy ( ε, δ ) -DP by basic composition in
Revised: Its output ˜ y K together with the output ˜ x K of Algorithm 4 satisfy ( ε, δ ) -DP by basic composition in

Original: Lemma 2.1.
Revised: Lemma 2.1.

Original: The guarantees are shown in the following theorem with detailed proofs in Appendix B.
Revised: The guarantees are shown in the following theorem with detailed proofs in Appendix B.

Original: Theorem 4.5.
Revised: Theorem 4.5.

Original: Let Assumption 4.1 hold.
Revised: Let Assumption 4.1 hold.

Original: Suppose max {∥ x ∥ , ∥ y ∥} ≤ D for all x ∈ X and y ∈ Y .
Revised: Suppose max {∥ x ∥ , ∥ y ∥} ≤ D for all x ∈ X and y ∈ Y .

Original: Then the composition of Algorithm 4 and 5 is ( ε, δ ) -DP
Revised: Then the composition of Algorithm 4 and 5 is ( ε, δ ) -DP

Original: and the output (˜ x K , ˜ y K ) satisfies the followingbound on the population weak duality gap:
Revised: and the output (˜ x K , ˜ y K ) satisfies the followingbound on the population weak duality gap:

Original: [Equation]
Revised: [Equation]

Original: when setting µ = ( L/D ) max (cid:110) 2 / √ n, 13 log n  (cid:112) d log(5 /δ ) / ( nε ) (cid:111) , where d = max { d x , d y } .
Revised: when setting µ = ( L/D ) max (cid:110) 2 / √ n, 13 log( n ) (cid:112) d log(5 /δ ) / ( nε ) (cid:111) , where d = max { d x , d y } .

Original: Remark 4.
Revised: Remark 4.

Original: The gradient complexity of our algorithm is the total complexity of A to solve the smooth

Original: SC-SC saddle point problem at each phase.

Original: By Remark 3, we can set 1 /γ =
Revised: By Remark 3, we can set 1 /γ =

Original: 16 µ ¯ n 2 / ( δL 2 ) to satisfythe requirements of the output.
Revised: 16 µ ¯ n 2 / ( δL 2 ) to satisfy the requirements of A ineach phase.

Original: For example, if we use Extragradient [40], the total complexity is2 (cid:80)
Revised: For example, if we use Extragradient [42], the total complexity is 2 (cid:80)

Original: Kk
Revised: Kk

Original: =1 O ((¯ n ( ℓ + µ k ) /µ )
Revised: =1 O ((¯ n ( ℓ +µ k ) /µ )

Original: log(¯ n/δ ))
Revised: log(¯ n/δ ))

Original: =
Revised: =

Original: O (( n 3 / 2 ℓ + n 2 ) log( n/δ )) .
Revised: O (( n 3 / 2 ℓ + n 2 ) log( n/δ )) .

Original: After similar calculations, thecomplexity becomes O ((n + n √ ℓ + nℓ + n 5 / 4 ) log( n/δ ))
Revised: After similar calculations, the complexity becomes O (( nℓ + n 5 / 4 ) log( n/δ ))

Original: if A is AL-SVRE [31] or Catalyst-AccSVRG [43] noticing that the condition number ( ℓ + µ k ) /µ can be as large as O ( n ) .
Revised: if A is AL-SVRE [33] or Catalyst-Acc-SVRG [46] noticing that thecondition number ( ℓ + µ k ) /µ can be as large as O ( n ) .

Revised: Here, the dependence on D/L is omitted.

Original: Remark 5.
Revised: Remark 5.

Original: To achieve near-linear time-complexity, we utilize the special structures of our regularizedproblems in Algorithm 4 and 5.
Revised: To achieve near-linear time-complexity, we utilize the special structures of our regularizedproblems in Algorithm 4 and 5.

Original: (i)
Revised: (i)

Original: Objective functions of the regularized problems are separablewith different smoothness parameters w.r.t.
Revised: Objective functions of the regularized problems are separablewith different smoothness parameters w.r.t.

Original: x and y .
Revised: x and y .

Original: If we use the algorithm in Jin et al.
Revised: Applying the algorithm in Jin et al.

Original: [23] thatis optimized for such special case, we can achieve the overall complexity O (( n + nℓ ) log( n/δ )) 
Revised: [24] that isoptimized for such special case, we can achieve the overall complexity O (( n + nℓ ) log( n/δ )) .

Original: .(ii) The regularization terms with potentially large smoothness parameters are prox-friendly 1 .
Revised: (ii) Theregularization terms with potentially large smoothness parameters are prox-friendly 2 .

Original: Thuswe can use SVRG/SAGA and its accelerated version [36] which leverage this structure, and thecomplexity only depends on the smoothness parameter of f ( x, y ; ξ ) .
Revised: SVRG/SAGAand its accelerated version [38] which leverage this property can be used, and the complexity onlydepends on the smoothness parameter of f ( x, y ; ξ ) .

Original: The total complexity is O (( n +nℓ 2 ) log( n/δ )) if A is SVRG/SAGA, and O (( n + nℓ ) log( n/δ )) if A is Acc-SVRG/SAGA, noticingthat now the condition number ℓ/µ is only O ( √ nℓ ) .
Revised: The total complexity is O (( n + nℓ 2 ) log( n/δ ))for SVRG/SAGA, and O (( n + nℓ ) log( n/δ )) for Acc-SVRG/SAGA, noticing that now the conditionnumber ℓ/µ is only O ( √ nℓ ) .

Original: More details are provided in Appendix B.
Revised: More details are provided in Appendix B.

Original: Algorithm 4 can be extended to the convex–strongly-concave case or the strongly-convex–concavecase, where near-linear time algorithmswith near-optimal guarantees are also possible. We providemore discussions in Appendix B.
Revised: Algorithm 4 can be extended to the smooth C-SC case or the SC-C case, where near-linear timenear-optimal algorithms are also provided (see Appendix B).

Original: As in the convex case in Section 3.2, the smoothness assumptionhere is not necessary for the  utility bound, but makes it possible to apply more efficient algorithms.
Revised: According to the lower-bound discussedin [10], our utility guarantee is optimal up to logarithmic terms.

Original: Although the primal problem min x ∈X Φ( x ) ≜ max y ∈Y F ( x, y ) is not necessarily smooth when eachf ( x, y ; ξ ) is smooth and convex-concave, we provide several instances of our flexible framework thatachieve near-optimal rates with near-linear time-complexity.
Revised: Although the primal problemmin x ∈X Φ( x ) ≜ max y ∈Y F ( x, y ) is not necessarily smooth when f ( x, y ; ξ ) is smooth and convexconcave, we provide several instances of the flexible framework that achieve near-optimal rates innear-linear time.

Original: This improves upon current results for DP-SMO [10, 44] and provides a new example where near-linear time algorithms exist for nonsmooth DP-SCO.
Revised: This improves upon current results for DP-SMO [10, 47] and gives a new examplewhere near-linear time algorithms are available for nonsmooth DP-SCO.

Original: It remains an open question whether linear-time algorithms with optimal utility bound canbe obtained  for general nonsmoothconvex DP minimization problems.
Revised: It remains an open questionwhether optimal utility bound can be obtained in linear time for general nonsmooth problems.

Original:  Conclusion
Revised:  Conclusion

Original: We provide a general framework for both smooth DP-SCO and DP-SMO problems with near-optimalprivacy-utility trade-offs.
Revised: We provide a general framework for both smooth DP-SCO and DP-SMO problems with near-optimalprivacy-utility trade-offs.

Original: The flexible framework allows to bring various off-the-shelf, fast convergentnon-private optimization algorithms into the DP domain.
Revised: The flexible framework allows to bring various off-the-shelf, fast convergentnon-private optimization algorithms into the DP domain.

Original: Using the framework, we enrich the class ofnear-linear time algorithms for smooth DP-SCO and provide the first near-linear time algorithms forsmooth DP-SMO.
Revised: Using the framework, we enrich the class ofnear-linear time algorithms for smooth DP-SCO and provide the first near-linear time algorithms forsmooth DP-SMO.

Original: For future work, it is interesting to study whether the logarithmic terms in gradientcomplexity and final utility bound of our algorithms can be further removed.
Revised: For future work, it is interesting to study whether the logarithmic terms in gradientcomplexity and final utility bound of our algorithms can be further removed.

Original: Another direction is toderive simpler private algorithms for smooth convex-concave DP-SMO based on the special structureof the regularized problems.
Revised: Another direction is toderive simpler private algorithms for smooth convex-concave DP-SMO based on the special structureof the regularized problems.

Original: Moreover, we believe our framework also opens the door to simplerderivatives with possibly improved complexities for nonsmooth DP-SCO, nonsmooth DP-SMO, andeven more general forms of optimization problems such as variational inequalities and games.
Revised: Moreover, we believe our framework also opens the door to simplerderivatives with possibly improved complexities for nonsmooth DP-SCO, nonsmooth DP-SMO, andeven more general forms of optimization problems such as variational inequalities and games.

Original: Wewill leave these for future investigation.
Revised: Wewill leave these for future investigation.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes] See Section 3 for contributions in DP-SCO and Section 4 for contributions in DP-SMO.
Revised: [Yes] See Section 3 for contributions in DP-SCO and Section4 for contributions in DP-SMO.

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] See Section 5.
Revised: [Yes] See Section 5.(c)

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: Did you discuss any potential negative societal impacts of your work?

Original: [N/A]
Revised: [N/A]

Original: This
Revised: This

Original: paper is purely theoretical and has no immediate societal impacts.
Revised: paper is purely theoretical and has no immediate societal impacts.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] See Assumption 3.1 and 3.2 for DP-SCO and Assumption 4.1 and 4.2 for DP-SMO.
Revised: [Yes] See Assumption3.1 and 3.2 for DP-SCO and Assumption 4.1 and 4.2 for DP-SMO.

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] See Appendix A for DP-SCO and Appendix B for DP-SMO.
Revised: [Yes] See Appendix A forDP-SCO and Appendix B for DP-SMO.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you report error bars (e g., with respect to the random seed after running experi-
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experi-

Original: ments multiple times)?
Revised: ments multiple times)?

Original: [N/A]
Revised: [N/A]

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., type

Original: of GPUs, internal cluster, or cloud provider)?
Revised: of GPUs, internal cluster, or cloud provider)?

Original: [N/A]
Revised: [N/A]

Original: If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]
Revised: [N/A]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A

Original:  (b) Did you describe any potential participant risks, with links to Institutional Review 
Revised: [Table] (b) Did you describe any potential participant risks, with links to Institutional Review [Table]

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original:  (c) Did you include the estimated hourly wage paid to participants and the total amount 
Revised: [Table] (c) Did you include the estimated hourly wage paid to participants and the total amount [Table]

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Revised: [Table]