Original: Bring Your Own Algorithm for Optimal Differentially Private Stochastic Minimax Optimization
Revised: Bring Your Own Algorithm for Optimal Differentially Private Stochastic Minimax Optimization

Original: Anonymous Author(s)

Original: Affiliation

Original: Address email

Original: Abstract
Revised: Abstract

Original: We study differentially private (DP) algorithms for smooth stochastic minimaxoptimization, with stochastic minimization as a byproduct.
Revised: We study differentially private (DP) algorithms for smooth stochastic minimax optimization, with stochastic minimization as a byproduct.

Original: The holy grail of thesesettings is to guarantee the optimal trade-off between the privacy and the excesspopulation loss, using an algorithm with a linear time-complexity in the numberof training samples.
Revised: The holy grail of these settings is to guarantee the optimal trade-off between the privacy and the excess population loss, using an algorithm with a linear time-complexity in the number of training samples.

Original: We provide a general framework for solving differentiallyprivate stochastic minimax optimization (DP-SMO) problems, such that practitioners can bring their own base optimization algorithm and take it as a black-box toobtain the near-optimal privacy-loss trade-off.
Revised: We provide a general framework for solving differentially private stochastic minimax optimization (DP-SMO) problems, which enables the practitioners to bring their own base optimization algorithm and use it as a black-box to obtain the near-optimal privacy-loss trade-off.

Original: Our framework is inspired fromthe recently proposed Phased-ERM method [21] for nonsmooth differentially private stochastic convex optimization (DP-SCO), which exploits the stability of theempirical risk minimization (ERM) for the privacy guarantee.
Revised: Our framework is inspired from the recently proposed Phased-ERM method [22] for nonsmooth differentially private stochastic convex optimization (DP-SCO), which exploits the stability of the empirical risk minimization (ERM) for the privacy guarantee.

Original: The flexibility ofour approach enables us to sidestep the requirement that the base algorithm needsto have bounded sensitivity, and allows the use of sophisticated variance-reducedaccelerated methods to achieve near-linear time-complexity.
Revised: The ﬂ exibility of our approach enables us to sidestep the requirement that the base algorithm needs to have bounded sensitivity, and allows the use of sophisticated variance-reduced accelerated methods to achieve near-linear time-complexity.

Original: To the best of ourknowledge, these are the first near-linear time algorithms with near-optimal guarantees on the population duality gap for smooth DP-SMO, when the objective is(strongly-)convex–(strongly-)concave.
Revised: To the best of our knowledge, these are the ﬁ rst near-linear time algorithms with near-optimal guarantees on the population duality gap for smooth DP-SMO, when the objective is (strongly-)convex–(strongly-)concave.

Original: Additionally, based on our flexible framework, we enrich the family of near-linear time algorithms for smooth DP-SCO withnear-optimal privacy-loss trade-offs.
Revised: Additionally, based on our ﬂ exible framework, we enrich the family of near-linear time algorithms for smooth DP-SCO with the near-optimal privacy-loss trade-off.

Original:  Introduction
Revised:  Introduction

Original: Machine learning models are nowadays trained using large corpora of data samples collected frommany different entities, e g., from users of a large software service [27].
Revised: Machine learning models are nowadays trained using large corpora of data samples collected from many different entities, e.g., from users of a large software service [28].

Original: However, it has beenempirically shown that these models can be exploited to reveal private information about thesecontributing entities.
Revised: However, it has been empirically shown that these models can be exploited to reveal private information about these contributing entities.

Original: For example, Carlini et al.
Revised: For example, Carlini et al.

Original: [12] attacked the large language model, GPT- toreveal hundreds of verbatim text samples used to train these models.
Revised: [13] attacked the large language model, GPT-2, to reveal hundreds of verbatim text samples used to train these models.

Original: These attacks violate the privacyof the contributing entities, but naturally they expect that no private information about them can berevealed through these models.
Revised: These attacks violate the privacy of the contributing entities, but naturally they expect that no private information about them can be revealed through these models.

Original: Over the last decade, this expectation was even legislated into lawssuch as GDPR in EU [14].
Revised: Over the last decade, this expectation was even legislated into laws such as GDPR in EU [15].

Original: There are many mathematical frameworks formalizing this expectationof privacy, but the most widely accepted one is that of Differential Privacy (DP) [17].
Revised: There are many mathematical frameworks formalizing this expectation of privacy, but the most widely accepted one is that of Differential Privacy (DP) [18].

Original: With highprobability, models satisfying DP cannot be attacked by any adversary to identify that a particulartraining sample was used in its training.
Revised: With high probability, models satisfying DP cannot be attacked by any adversary to identify that a particular training sample was used in its training.

Original: Hence, DP provides any entity plausible deniability that theycontributed to the training set.
Revised: Hence, DP provides any entity plausible deniability that they contributed to the training set.

Original: However, optimization algorithms for training such models under DPrequire careful design choices to ensure privacy while preventing the degradation of convergence  speed and data efficiency.
Revised: However, optimization algorithms for training such models under DP require careful design choices to ensure privacy while preventing the degradation of convergence Information Processing speed and data ef ﬁ ciency.

Original: This has led to the burgeoning field of differentially private optimizationalgorithms [5, 6, 21], which considers stochastic convex minimization as the canonical problem.
Revised: This has led to the burgeoning ﬁ eld of differentially private optimization algorithms [6, 7, 22], which considers stochastic convex minimization as the canonical problem.

Original: For differentially private stochastic convex optimization (DP-SCO) with ( ε, δ ) -DP guarantees, theoptimal excess population risk is Θ(1 / ( µn ) + d log(1 /δ ) /
Revised: For differentially private stochastic convex optimization (DP-SCO) with ( ε , δ ) -DP guarantees, the optimal excess population risk is Θ (1 / ( µn ) + d log(1 / δ ) /

Original: ( µn 2 ε 2 )) for µ -strongly convex objectives,where n is the number of participants and d is the dimension of the variable.
Revised: ( µn 2 ε 2 )) for µ -strongly convex objectives, where n is the number of participants and d is the dimension of the variable.

Original: If the objective isalso smooth, this can be achieved with a linear-time ( ε, δ ) -DP algorithm
Revised: If the objective is also smooth, this can be achieved with a linear-time ( ε , δ ) -DP algorithm

Original: [21] using O ( n ) stochasticgradient evaluations.
Revised: [22] using O ( n ) stochastic gradient evaluations.

Original: This analysis critically relies on the concept of algorithmic stability [11], whichmeasures how much the population loss of an algorithm’s output changes when a single data pointin the input dataset is perturbed.
Revised: This analysis critically relies on the concept of algorithmic stability [12], which measures how much the population loss of an algorithm’s output changes when a single data point in the input dataset is perturbed.

Original: This is also known as sensitivity of an algorithm in DP, whichdetermines how much noise needs to be added in order to achieve ( ε, δ ) -DP
Revised: This is also known as sensitivity of an algorithm in DP, which determines how much noise needs to be added in order to achieve ( ε , δ ) -DP

Original: (see Definition 2).
Revised: (see De ﬁ nition 2).

Original: Wewill use stability and sensitivity interchangeably when referring to optimization algorithms.
Revised: We will use stability and sensitivity interchangeably when referring to optimization algorithms.

Original: As thesensitivity of stochastic gradient descent (SGD) is known [23], Feldman et al.
Revised: As the sensitivity of stochastic gradient descent (SGD) is known [24], Feldman et al.

Original: [21] was able to addan appropriate amount of noise—tailored to this sensitivity—to an SGD-based algorithm to achieveoptimal risk in linear time.
Revised: [22] were able to add an appropriate amount of noise—tailored to this sensitivity—to an SGD-based algorithm to achieve optimal risk in linear time.

Original: However, such tight analysis of sensitivity is generally intractable for morecomplex optimization routines that practitioners might want to use.
Revised: However, such tight analysis of sensitivity is generally intractable for more complex optimization routines that practitioners might want to use.

Original: Further, the stability analysis of
Revised: Further, the stability analysis of

Original: SGD only holds when the smoothness parameter ℓ of the problem is upper-bounded by ˜ O ( µn ) .
Revised: SGD only holds when the smoothness parameter ℓ of the problem is upper-bounded by ˜ O ( µn ) .

Original: Inthe first part of the paper, we show that we can achieve similar guarantees using a wider choice of,potentially more practical, “base” algorithms, without any restrictions on the smoothness parameter.
Revised: In the ﬁ rst part of the paper, we show that we can achieve similar guarantees using a wider choice of, potentially more practical, “base” algorithms, without any restrictions on the smoothness parameter.

Original: On another front, many emerging practical machine learning applications are formulated as stochasticminimax optimization problems, e.g., generative adversarial networks [22], adversarially robustmachine learning [34], and reinforcement learning [15].
Revised: On another front, many emerging practical machine learning applications are formulated as stochastic minimax optimization problems, e.g., generative adversarial networks [23], adversarially robust machine learning [35], and reinforcement learning [16].

Original: Designing DP algorithms for solving theseminimax problems is of paramount importance.
Revised: Designing DP algorithms for solving these minimax problems is of paramount importance.

Original: For example, private generative adversarial networksprovide a promising new direction to synthetic data generation [45], such as in networked time-seriesdata [31].
Revised: For example, private generative adversarial networks provide a promising new direction to synthetic data generation [47], such as in networked time-series data [32].

Original: Motivated by such applications, we study the differentially private stochastic minimaxoptimization (DP-SMO) problem of the form:
Revised: Motivated by such applications, we study the differentially private stochastic minimax optimization (DP-SMO) problem of the form:

Original: [Equation]
Revised: [Equation]

Original: where the objective f ( x, y ; ξ ) is smooth and convex-concave for any random vector ξ , and we aregiven access to n i.i.d.
Revised: where the objective f ( x, y ; ξ ) is smooth and convex-concave for any random vector ξ , and we are given access to n i.i.d.

Original: samples { ξ i } ni =1 .
Revised: samples { ξ i } ni =1 .

Original: In contrast to DP-SCO, where linear-time algorithms havebeen proposed to achieve optimal risk guarantees, existing private algorithms [10, 47] for DP-SMOachieving optimal guarantees have time-complexity which scales super-linearly in the number ofsamples n (summarized in Table 1).
Revised: In contrast to DP-SCO, where linear-time algorithms have been proposed to achieve optimal risk guarantees, existing private algorithms [11, 49] for DP-SMO achieving optimal guarantees have time-complexity which scales super-linearly in the number of samples n (summarized in Table 1).

Original: In the second part of this paper, we close this gap by introducinga new class of output perturbation-based DP algorithms for both strongly-convex–strongly-concaveand convex-concave settings, which can achieve near-optimal population risk bounds using ˜ O ( n )stochastic gradient computations, where ˜ O hides logarithmic factors.
Revised: In the second part of this paper, we close this gap by introducing a new class of output perturbation-based DP algorithms for both strongly-convex–strongly-concave and convex-concave settings, which can achieve near-optimal population risk bounds using ˜ O ( n ) stochastic gradient computations, where ˜ O hides logarithmic factors.

Original: One of the main bottlenecks for (i) using a wider variety of algorithms to solve the DP-SCO problem,and (ii) solving the DP-SMO problem using linear-time algorithms, is the lack of known stabilityresults for fast and sophisticated non-private algorithms, such as variance-reduced and acceleratedmethods.
Revised: One of the main bottlenecks for (i) using a wider variety of algorithms to solve the DP-SCO problem, and (ii) solving the DP-SMO problem using linear-time algorithms, is the lack of known stability results for fast and sophisticated non-private algorithms, such as variance-reduced and accelerated methods.

Original: One can sidestep the algorithm-specific stability requirement by utilizing the stability ofthe optimal solution to a strongly-convex and Lipschitz empirical risk minimization (ERM) problem[41, 49].
Revised: One can sidestep the algorithm-speci ﬁ c stability requirement by utilizing the stability of the optimal solution to a strongly-convex and Lipschitz empirical risk minimization (ERM) problem [42, 51].

Original: Particularly, if the output of an algorithm for such an ERM problem is close enough to itsempirical solution, then the sensitivity of the algorithm is automatically guaranteed.
Revised: Particularly, if the output of an algorithm for such an ERM problem is close enough to its empirical solution, then the sensitivity of the algorithm is automatically guaranteed.

Original: Exploiting thisobservation, Feldman et al.
Revised: Exploiting this observation, Feldman et al.

Original: [21] proposed the phased-ERM algorithm for nonsmooth DP-SCO, whichsolves a series of strongly-convex ERM subproblems to sidestep the stability analysis of SGD fornonsmooth functions.
Revised: [22] proposed the phased-ERM algorithm for nonsmooth DP-SCO, which solves a series of strongly-convex ERM subproblems to sidestep the stability analysis of SGD for nonsmooth functions.

Original: This algorithm achieves a quadratic time-complexity for nonsmooth DP-SCO.
Revised: This algorithm achieves a quadratic time-complexity for nonsmooth DP-SCO.

Original: We observe that when the problem is additionally smooth, or when it allows a smooth minimaxreformulation [36], there exist a plethora of fast algorithms to solve the resulting ERM subproblems.
Revised: We observe that when the problem is additionally smooth, or when it allows a smooth minimax reformulation [37], there exist a plethora of fast algorithms to solve the resulting ERM subproblems.

Original: Combining these algorithms with (phased) output perturbation gives rise to a class of near-optimalnear-linear 1 time-complexity private algorithms for both smooth DP-SCO and DP-SMO problems,without any additional effort on their stability analysis.
Revised: Combining these algorithms with (phased) output perturbation gives rise to a class of near-optimal near-linear 1 time-complexity private algorithms for both smooth DP-SCO and DP-SMO problems, without any additional effort on their stability analysis.

Original: Our contributions are summarized below:
Revised: Our contributions are summarized below:

Original: • We introduce a flexible framework for solving smooth DP-SMO and smooth DP-SCO problems,utilizing the (phased) output perturbation mechanism.
Revised: • We introduce a ﬂ exible framework for solving smooth DP-SMO and smooth DP-SCO problems, utilizing the (phased) output perturbation mechanism.

Original: The black-box framework enables us to bypassthe need to prove algorithm-specific stability and transform off-the-shelf optimization algorithms
Revised: The black-box framework enables us to bypass the need to prove algorithm-speci ﬁ c stability and transform off-the-shelf optimization algorithms

Original: [Table]
Revised: [Table]

Original: into DP algorithms with near-optimal guarantees.
Revised: into DP algorithms with near-optimal guarantees.

Original: This is attractive as there are currently no stabilityanalyses for accelerated [2, 38] and variance-reduced [25, 38] algorithms for both SMO and SCO.•
Revised: This is attractive as there are currently no stability analyses for accelerated [2, 39] and variance-reduced [26, 39] algorithms for both SMO and SCO.

Original:  Using the framework
Revised: • Using the framework

Original: [Table]
Revised: [Table]

Original: we provide the first near-linear time private algorithms for smooth DP-SMOwith near-optimal bound on the population duality gap, under both strongly-convex–strongly-concave and convex-concave cases (see Table 1).
Revised: we provide the ﬁ rst near-linear time private algorithms for smooth DP-SMO with near-optimal bound on the population duality gap, under both strongly-convex–strongly-concave and convex-concave cases (see Table 1).

Original: Among other things, this implies that if a (primal) nonsmoothminimization problem can be reformulated as a smooth convex-concave minimax optimization, thenwe can solve it in near-linear time instead of the best known super-linear time [4, 29].
Revised: Among other things, this implies that if a (primal) nonsmooth minimization problem can be reformulated as a smooth convex-concave minimax optimization problem, then we can solve it in near-linear time instead of the best known super-linear time [4, 30].

Original: In priorwork, linear-time DP algorithms with optimal rates for solving nonsmooth  objectives only existed forgeneralized linear losses [8].• The framework also enriches the cohort of near-linear time near-optimal private algorithms forsmooth DP-SCO settings, that only contained SGD previously.
Revised: In prior work, near-linear time DP algorithms with optimal rates for solving nonsmooth convex objectives only existed for generalized linear losses [9]. • The framework also enriches the cohort of near-linear time near-optimal private algorithms for smooth DP-SCO settings, which only contained SGD previously.

Original: Moreover, existing optimal DPalgorithms for smooth DP-SCO rely on a stability result of SGD, whichholds only  for a restrictedrange of the smoothness parameter [23].
Revised: Moreover, existing optimal DP algorithms for smooth DP-SCO rely on a stability result of SGD, which only holds for a restricted range of the smoothness parameter [24].

Original: Such restrictions are avoided in our framework.
Revised: Such restrictions are avoided in our framework.

Original: Related Works Differentially private optimization has been an active research field for the past few years, and early works focused on the empirical problems [5, 48, 43].
Revised: Related Works: Differentially private optimization has been an active research ﬁ eld for the past few years, and early works focused on the empirical problems [6, 46, 50, 44].

Original: In addition to the output perturbation used in this paper, many existing works applied gradient perturbation to guarantee privacy.
Revised: In addition to the output perturbation used in this paper, many existing works applied gradient perturbation to guarantee privacy.

Original: This method adds noise to each iteration of the algorithm and then applies moments accountant [1] or advanced composition [26] to analyze the overall privacy.
Revised: This method adds noise to each iteration of the algorithm and then applies moments accountant [1] or advanced composition [27] to analyze the overall privacy.

Original: Although gradient perturbation does notneed the smoothness or convexity assumptions of the function and works for most iterative algorithms, it requires larger mini-batches [6] or longer training time [7] if one resorts to the privacy amplificationvia subsampling [9] to reduce the DP noise, resulting in super-linear gradient queries in existingmethods [6, 47].
Revised: Although gradient perturbation does not need the smoothness or convexity assumptions of the function and works for most iterative algorithms, it requires larger mini-batches [7] or longer training time [8] if one resorts to the privacy ampli ﬁ cation via subsampling [10] to reduce the DP noise, resulting in super-linear gradient queries in existing methods [7, 49].

Original: Here we mainly review previous results that achieve the optimal utility bound
Revised: Here we mainly review previous results that achieve the optimal utility bound

Original: O (1 / √ n + (cid:112) d log(1 /δ ) / ( nε )) on the population loss—according to a lower-bound in Bassily et al.[5]—when solving a d dimensional DP-SCO problem using n samples with ( ε, δ ) -DP guarantees.
Revised: O (1 / √ n + p d log(1 / δ ) / ( n ε )) on the population loss—according to a lower-bound in Bassily et al. [6]—when solving a d dimensional DP-SCO problem using n samples with ( ε , δ ) -DP guarantees.

Original: In the smooth convex case, Bassily et al.
Revised: In the smooth convex case, Bassily et al.

Original: [6] is the first to derive optimal rates for DP-SCO withcomplexity O ( n 3 / 2 √ ε ) by gradient perturbation and stability of SGD [23].
Revised: [7] is the ﬁ rst to derive optimal rates for DP-SCO with complexity O ( n 3 / 2 √ ε ) by gradient perturbation and stability of SGD [24].

Original: Feldman et al.
Revised: Feldman et al.

Original: [21]provided two SGD-based linear-time algorithms: one uses privacy amplification by iteration [20] thatonly works for contractive updates; the second one uses phased output perturbation and stability of
Revised: [22] provided two SGD-based linear-time algorithms: one uses privacy ampli ﬁ cation by iteration [21] that only works for contractive updates; the second one uses phased output perturbation and stability of

Original: SGD.
Revised: SGD.

Original: In the nonsmooth convex case, Bassily et al.
Revised: In the nonsmooth convex case, Bassily et al.

Original: [7] established the stability of SGD for nonsmoothfunctions and obtained a quadratic-time algorithm.
Revised: [8] established the stability of SGD for nonsmooth functions and obtained a quadratic-time algorithm.

Original: Feldman et al.
Revised: Feldman et al.

Original: [21] leveraged the stability of ERMsolutions [41] and achieved O ( n 2 log(1 /δ )) complexity with phased output perturbation.
Revised: [22] leveraged the stability of ERM solutions [42] and achieved O ( n 2 log(1 / δ )) complexity with phased output perturbation.

Original: Based onthis phased framework, Asi et al.
Revised: Based on this phased framework, Asi et al.

Original: [4] and Kulkarni et al.
Revised: [4] and Kulkarni et al.

Original: [29] used gradient perturbation and improvedthe complexity to O (min { n 3 / 2 , n 2 / √ d } ) and O (min { n 5 / 4 d 1 / 8 , n 3 / 2 /d 1 / 8 } ) respectively; Asi et al.[3] introduced a hypothetical linear-time algorithm assuming the existence of a low-biased estimator.
Revised: [30] used gradient perturbation and improved the complexity to O (min { n 3 / 2 , n 2 / √ d } ) and O (min { n 5 / 4 d 1 / 8 , n 3 / 2 /d 1 / 8 } ) respectively; Asi et al. [3] introduced a hypothetical linear-time algorithm assuming the existence of a low-biased estimator.

Original: Moreover, Bassily et al.
Revised: Moreover, Bassily et al.

Original: [8] gave a linear-time algorithm for  generalized linear losses using phased SGD [21] with smoothing techniques.
Revised: [9] gave a near-linear time algorithm for nonsmooth convex generalized linear losses using phased SGD [22] with smoothing techniques [37].

Original: To the best of our knowledge, only few papers studied DP-SMO and all of them used gradientperturbation to guarantee privacy.
Revised: To the best of our knowledge, only few papers studied DP-SMO and all of them used gradient perturbation to guarantee privacy.

Original: Boob and Guzmán [10] analyzed stability of Extragradient [42] and proximal point methods [40] for smooth convex-concave functions, and their DP versions run withtime O ( n 2 ) and ˜ O ( n 3 / 2 ) respectively.
Revised: Boob and Guzmán [11] analyzed stability of Extragradient [43] and proximal point methods [41] for smooth convex-concave functions, and their DP versions run with time O ( n 2 ) and ˜ O ( n 3 / 2 ) respectively.

Original: Yang et al.
Revised: Yang et al.

Original: [47] used the stability of SGDA [30] and obtained
Revised: [49] used the stability of SGDA [31] and obtained

Original: DP algorithms with complexity O ( n 3 / 2 √ ε ) for the smooth convex-concave case and O ( n 2 ) for thenonsmooth case, mirroring the guarantees of SGD for DP-SCO [6, 7].
Revised: DP algorithms with complexity O ( n 3 / 2 √ ε ) for the smooth convex-concave case and O ( n 2 ) for the nonsmooth case, mirroring the guarantees of SGD for DP-SCO [7, 8].

Original: Kang et al.
Revised: Kang et al.

Original: [28] only focusedon stability and generalization analysis of gradient perturbed SGDA and provided high-probabilityresults.
Revised: [29] only focused on stability and generalization analysis of gradient perturbed SGDA and provided high-probability results.

Original: See Table 1 for a brief comparison.
Revised: See Table 1 for a brief comparison.

Original: The idea of using the stability of ERM for smooth DP-SCO was also  exploited in Lowy and
Revised: The idea of using the stability of ERM for smooth convex optimization has also been exploited before.

Original: Razaviyayn [32].
Revised: Attia and Koren [5] developed a black-box framework for smooth convex objectives that produces uniformly-stable algorithms while maintaining fast convergence rates.

Original: However, they restricted to output perturbation and only provided near-linear timenear-optimal algorithms for smooth strongly-convex losses.
Revised: Lowy and Razaviyayn [33] considered DP-SCO by output perturbation but only provided near-linear time near-optimal algorithms for smooth strongly-convex losses.

Original: For smooth convex case, they achieved asub-optimal rateO (1 / √ n + ( (cid:112)
Revised: For smooth convex case, they achieved a sub-optimal rate

Original:  d log(1 /δ ) / ( nε )) 2 / 3 ) in near-linear time.
Revised: O (1 / √ n + ( p d log(1 / δ ) / ( n ε )) 2 / 3 ) in near-linear time.

Original: In the setting of smoothstrongly-convex–(strongly-)concave DP-SMO, they directly applied the DP-SCO reformulation ofthe primal function to obtain the final guarantees.
Revised: In the setting of smooth strongly-convex– (strongly-)concave DP-SMO, they directly utilized its DP-SCO reformulation in the primal function to obtain the ﬁ nal guarantees.

Original: Therefore, their utility bound is simply on theprimal risk, which is weaker than the primal-dual gap considered in this work.
Revised: Therefore, their utility bound is simply on the primal risk, which is weaker than the primal-dual gap considered in this work.

Original: The algorithms run innear-linear time for the strongly-concave case and super-linear time O ( n 5 / 2 ) for the concave case.
Revised: The algorithms run in near-linear time for the strongly-concave case and super-linear time O ( n 5 / 2 ) for the concave case.

Original: Incontrast, we provide near-optimal near-linear time algorithms in all aforementioned settings.
Revised: In contrast, we provide near-optimal near-linear time algorithms in all aforementioned settings.

Original: Notations: We use ∥·∥ for the Euclidean norm of a vector and |·| for the absolute value or thecardinality of a set.
Revised: Notations: We use ∥ · ∥ for the Euclidean norm of a vector and |·| for the absolute value or the cardinality of a set.

Original: A function g : R d → R is L -Lipschitz if | g ( x 1 ) − g ( x 2 ) | ≤ L ∥ x 1 − x 2 ∥for x 1 , x 2 in the domain of g .
Revised: A function g : R d → R is L -Lipschitz if | g ( x 1 ) − g ( x 2 ) | ≤ L ∥ x 1 − x 2 ∥ for x 1 , x 2 in the domain of g .

Original: A function h : R d → R is ℓ -smooth if it is differentiable andh ( x 2 )
Revised: A function h : R d → R is ℓ -smooth if it is differentiable and h ( x 2 )

Original: ≤ h ( x 1 ) + ∇ h ( x 1 ) ⊤ ( x 2 − x 1 ) + ( ℓ/ 2) ∥ x 1 − x 2 ∥ 2 .
Revised: ≤ h ( x 1 ) + ∇ h ( x 1 ) ⊤ ( x 2 − x 1 ) + ( ℓ / 2) ∥ x 1 − x 2 ∥ 2 .

Original: A function p : R d → R is convex ifp ( αx 1 + (1 − α ) x 2 ) ≤ αp ( x 1 ) + (1 − α ) p ( x 2 ) for all α ∈ [0 , 1] , and p is µ -strongly convex ifp ( x ) − ( µ/ 2)
Revised: A function p : R d → R is convex if p ( α x 1 + (1 − α ) x 2 ) ≤ α p ( x 1 ) + (1 − α ) p ( x 2 ) for all α ∈ [0 , 1] , and p is µ -strongly convex if p ( x ) − ( µ/ 2)

Original: ∥ x ∥ 2 is convex with µ > 0 .
Revised: ∥ x ∥ 2 is convex with µ > 0 .

Original: A function q : R d → R is concave if − q is convex andµ -strongly concave if − q is µ -strongly convex.
Revised: A function q : R d → R is concave if − q is convex and µ -strongly concave if − q is µ -strongly convex.

Original: For a vector x ∈ R d , the notation x + N (0 , σ 2 I d )means x + z for a random vector z ∼ N (0 , σ 2 I d ) sampled from the Gaussian distribution.
Revised: For a vector x ∈ R d , the notation x + N (0 , σ 2 I d ) means x + z for a random vector z ∼ N (0 , σ 2 I d ) sampled from the Gaussian distribution.

Original:  Preliminaries
Revised:  Preliminaries

Original: We first provide some background on differential privacy and stochastic minimax optimization.
Revised: We ﬁ rst provide some background on differential privacy and stochastic minimax optimization.

Original: 2.1 Differential Privacy
Revised: 2.1 Differential Privacy

Original: Differential Privacy (DP), introduced in Dwork et al.
Revised: Differential Privacy (DP), introduced in Dwork et al.

Original: [17], measures privacy leakage of an algorithm.
Revised: [18], measures privacy leakage of an algorithm.

Original: Definition 1.
Revised: De ﬁ nition 1.

Original: For two datasets S = { ξ i } ni =1 and S ′
Revised: For two datasets S = { ξ i } ni =1 and S ′

Original: = { ξ ′ i } ni =1 , we say the pair ( S, S ′ ) is neighboringif max {| S \ S ′
Revised: = { ξ ′ i } ni =1 , we say the pair ( S, S ′ ) is neighboring if max {| S \ S ′

Original: | , | S ′ \ S |} = 

Revised: | , | S ′ \ S |} =

Original: and we denote neighboring datasets with S ∼ S ′ .
Revised: and we denote neighboring datasets with S ∼ S ′ .

Original: For an algorithm Aand some privacy parameters ε > 0 and δ ∈ (0 , 1) , we say A satisfies ( ε, δ ) - differential privacy if152 P ( A ( S ) ∈
Revised: For an algorithm A and some privacy parameters ε > 0 and δ ∈ (0 , 1) , we say A satis ﬁ es ( ε , δ ) - differential privacy if

Original:  A ) ≤ e ε P
Revised: P ( A ( S ) ∈ A ) ≤ e ε P

Original: ( A ( S ′ )
Revised: ( A ( S ′ )

Original: ∈ A )
Revised: ∈ A )

Original: + δ for all S ∼ S ′ and all subset A in the range of A .
Revised: + δ for all S ∼ S ′ and all subset A in the range of A .

Original: In this work, we focus on the settings when ε ∈ (0 , 1) and δ ∈ (0 , 1 /n ) given dataset of size n .
Revised: In this work, we focus on the settings when ε ∈ (0 , 1) and δ ∈ (0 , 1 /n ) given dataset of size n .

Original: Sensitivity is an important concept that makes designing ( ε, δ )
Revised: Sensitivity is an important concept that makes designing ( ε , δ )

Original: -DP mechanisms straightforward.
Revised: -DP mechanisms straightforward.

Original: Definition 2.
Revised: De ﬁ nition 2.

Original:  A  randomized algorithmA operating on S and outputting a vector in R d  has sensitivity∆ A := sup S ∼ S 
Revised: Let A be some randomized algorithm operating on S and outputting a vector in R d . If A has sensitivity ∆ A := sup S ∼ S ′

Original: ′ ∥A ( S ) − A ( S ′ )
Revised: ∥ A ( S ) − A ( S ′ )

Original: ∥ with probability at least 1 − δ . For this algorithm, the Gaussianmechanism outputs A ( S ) + N (0 , (∆ A (cid:112) 2 log(1 .
Revised: ∥ with probability at least 1 − δ , then the Gaussian mechanism outputs A ( S ) + N (0 , ( ∆ A p 2 log(1 .

Original: 25 /δ ) /ε ) 2 I d ) and achieves ( ε, 2 δ ) -DP 
Revised: 25 / δ ) / ε ) 2 I d ) and achieves ( ε , 2 δ ) -DP [17, 22].

Original: The following basic composition rule of differential privacy will be used in the analysis.
Revised: The following basic composition rule of differential privacy will be used in the analysis.

Original: Lemma 2.1.
Revised: Lemma 2.1.

Original: If A 1 is ( ε 1 , δ 1 ) -DP
Revised: If A 1 is ( ε 1 , δ 1 ) -DP

Original: and A 2 is ( ε 2 , δ 2 )
Revised: and A 2 is ( ε 2 , δ 2 )

Original: -DP, then ( A 1 , A 2 ) is ( ε 1 + ε 2 , δ 1 + δ 2 ) -DP[16].
Revised: -DP, then ( A 1 , A 2 ) is ( ε 1 + ε 2 , δ 1 + δ 2 ) -DP

Original: For a sequence of interactive queries {A k } Kk =1 each satisfying ( ε k , δ k ) -DP and operating on a subset S k , if S k ’s are disjoint then the composition ( A 1 ( S 1 ) , A 2 ( S 2 ) , . . .
Revised: For a sequence of interactive algorithms {A k } Kk =1 each satisfying ( ε k , δ k ) -DP and operat- ing on a subset S k , if S k ’s are disjoint then the composition ( A 1 ( S 1 ) , A 2 ( S 2 ) , . . .

Original: , A K ( S K )) is(max k ∈ [ K ] ε k , max k ∈
Revised: , A K ( S K )) is (max k ∈ [ K ] ε k , max k ∈

Original: [ K ] δ k ) -DP
Revised: [ K ] δ k ) -DP

Original: (known as parallel composition in McSherry [35]).
Revised: (known as parallel composition in McSherry [36]).

Revised: Input: Dataset S = { ξ i } ni =1 , algorithm A , DP parameters ( ε , δ ) , strong-convexity parameter µ .

Revised: 1: Run the algorithm A on the smooth strongly-convex ﬁ nite-sum problem min x ∈ X ˆ F S ( x ) =

Revised: (1 /n )

Revised: P ni =1 f ( x ; ξ i ) to obtain the output A ( S ) such that ∥ A ( S ) − ˆ x ∗ S ∥ ≤ L/ ( µn ) with probability at least 1 − δ / 4 , where ˆ x ∗ S = arg

Original: 2.2 Stochastic Minimax Optimization
Revised: min x ∈ X ˆ F S ( x ) is the empirical optimal solution. Output: ˜ x = A ( S ) + N (0 , σ 2 I d ) with σ = 4 L p 2 log(2 . 5 / δ ) /

Revised: ( µn ε ) .

Revised: 2.2 Stochastic Minimax Optimization

Original: The stochastic minimax (a.k.a.
Revised: The stochastic minimax (a.k.a.

Original: saddle point) optimization problem has the form:
Revised: saddle point) optimization problem has the form:

Original: [Equation]
Revised: [Equation]

Original: where F is the population-level expectation of the stochastic continuous objective f ( · , · ; ξ ) :
Revised: where F is the population-level expectation of the stochastic continuous objective f ( · , · ; ξ ) :

Original: R d x ×
Revised: R d x ×

Original: R d y → R with closed convex domains X ⊂ R d x and Y ⊂ R d y whose stochasticity is captured by 
Revised: R d y → R with closed convex domains X ⊂ R d x and Y ⊂ R d y whose stochasticity is captured by the random vector ξ .

Original: Input: Dataset S = { ξ i } ni =1 , algorithm A , DP parameters ( ε, δ ) , strongly-convex parameter µ .

Original: x ξ i obtain the output A ( S ) such that ∥A ( S ) − ˆ x ∗ S ∥ ≤ L/ ( µn ) with probability at least 1 − δ/ 4 ,

Original: ˆ x ∗ S = arg min x ∈X ˆ F S ( x ) is the empirical optimal solution.

Original: Output: d log δ ) /

Original: the random vector ξ .

Original: We are interested in the population saddle point ( x ∗ , y ∗ ) ∈
Revised: We are interested in the population saddle point ( x ∗ , y ∗ ) ∈

Original: X × Y of the aboveproblem such that F ( x ∗ , y ) ≤ F ( x ∗ , y ∗ ) ≤ F ( x, y ∗ ) for all ( x, y ) ∈
Revised: X × Y of the above problem such that F ( x ∗ , y ) ≤ F ( x ∗ , y ∗ ) ≤ F ( x, y ∗ ) for all ( x, y ) ∈

Original: X × Y .
Revised: X × Y .

Original: For some randomized algorithm with output (˜ x, ˜ y ) , we measure its convergence  by the population strong duality gap E [max y ∈Y F (˜ x, y ) − min x ∈X F ( x, ˜ y )] or the population weak duality gapmax y ∈Y E
Revised: For some randomized algorithm with output (˜ x, ˜ y ) , we measure its convergence rates by the population strong duality gap E [max y ∈ Y F (˜ x, y ) − min x ∈ X F ( x, ˜ y )] or the population weak duality gap max y ∈ Y E

Original: [ F (˜ x, y )] − min x ∈X E [ F ( x, ˜ y )] .
Revised: [ F (˜ x, y )] − min x ∈ X E [ F ( x, ˜ y )] .

Original: Note that both duality gaps are always larger than 0 , anda deterministic (˜ x, ˜ y ) is the saddle point if and only if the  gaps are 0 .
Revised: Note that both duality gaps are always larger than 0 , and a deterministic (˜ x, ˜ y ) is the saddle point if and only if the duality gaps are 0 .

Original: In practice, we usually do not have access to the distribution P ξ of the random vector ξ .
Revised: In practice, we usually do not have access to the distribution P ξ of the random vector ξ .

Original: Instead weare given a dataset S = { ξ i } ni =1 with n random vectors independently sampled from the distribution P ξ .
Revised: Instead we are given a dataset S = { ξ i } ni =1 with n random vectors independently sampled from the distribution P ξ .

Original: We define the empirical minimax optimization problem as
Revised: We de ﬁ ne the empirical minimax optimization problem as

Original: [Equation]
Revised: [Equation]

Original: (cid:88)

Original: [Equation]

Original: =

Original: [Equation]

Original: f ( x

Original: [Equation] y [Equation] Similarly, we can define the empirical saddle point (ˆ x ∗ S , ˆ y ∗ S ) and the empirical duality gap w.r.t.
Revised: Similarly, we can de ﬁ ne the empirical saddle point (ˆ x ∗ S , ˆ y ∗ S ) and the empirical duality gap w.r.t.

Original: theempirical function ˆ F S .
Revised: the empirical function ˆ F S .

Original: Zhang et al.
Revised: Zhang et al.

Original: [49] established the stability and generalization properties of theempirical saddle point, which is essential to the design of our DP-SMO algorithms.
Revised: [51] established the stability and generalization properties of the empirical saddle point, which is essential to the design of our DP-SMO algorithms.

Original:  Differentially Private Stochastic Convex Optimization
Revised:  Differentially Private Stochastic Convex Optimization

Original: As a warm-up, we start with a simpler problem of differentially private stochastic convex optimization(DP-SCO) to showcase our main ideas.
Revised: As a warm-up, we start with a simpler problem of differentially private stochastic convex optimization (DP-SCO) to showcase our main ideas.

Original: We consider the following stochastic optimization:
Revised: We consider the following stochastic optimization:

Original: [Equation]
Revised: [Equation]

Original: where the stochastic function f ( · , ξ ) : R d → R is defined on a convex domain X ⊂ R d and ξ is arandom vector from an unknown distribution P ξ .
Revised: where the stochastic function f ( · , ξ ) : R d → R is de ﬁ ned on a convex domain X ⊂ R d and ξ is a random vector from an unknown distribution P ξ .

Original: Given a dataset S with n i.i.d.
Revised: Given a dataset S with n i.i.d.

Original: samples from P ξ , wedevelop a generic output perturbation framework with both privacy and population loss guarantees.
Revised: samples from P ξ , we develop a generic output perturbation framework with both privacy and population loss guarantees.

Original: 3.1 Near-Linear Time Algorithms for Smooth Strongly-Convex Functions
Revised: 3.1 Near-Linear Time Algorithms for Smooth Strongly-Convex Functions

Original: First we study the case when the objective function is strongly-convex with the following assumptions.
Revised: First we study the case when the objective function is strongly-convex with the following assumptions.

Original: Assumption 3.1.
Revised: Assumption 3.1.

Original: For any ξ , the function f ( x ; ξ ) is L -Lipschitz, ℓ -smooth, and convex on the closedconvex domain X ⊂ R d .
Revised: For any ξ , the function f ( x ; ξ ) is L -Lipschitz, ℓ -smooth, and convex on the closed convex domain X ⊂ R d .

Original: Assumption 3.2.
Revised: Assumption 3.2.

Original: For any ξ , f ( x ; ξ ) satisfies Assumption 3.1 and it is µ -strongly convex on X .
Revised: For any ξ , f ( x ; ξ ) satis ﬁ es Assumption 3.1 and it is µ -strongly convex on X .

Original: Shalev-Shwartz et al.
Revised: Shalev-Shwartz et al.

Original: [41] proved that the empirical optimal solution has bounded stability if theobjective f ( · ; ξ ) is strongly-convex and Lipschitz w.r.t.
Revised: [42] proved that the empirical optimal solution has bounded stability if the objective f ( · ; ξ ) is strongly-convex and Lipschitz w.r.t.

Original: its domain (see Lemma A.2 in the appendix).
Revised: its domain (see Lemma A.2 in the appendix).

Original: As long as the output of some algorithm A is close enough to the empirical solution, we can showthat A has bounded sensitivity, and thus standard Gaussian mechanism (Definition 2) can be appliedto ensure differential privacy.
Revised: As long as the output of some algorithm A is close enough to the empirical solution, we can show that A has bounded sensitivity, and thus standard Gaussian mechanism (De ﬁ nition 2) can be applied to ensure differential privacy.

Original: This is formalized in Algorithm 1, where A is any algorithm for solvingsmooth strongly-convex finite-sum minimization problems.
Revised: This is formalized in Algorithm 1, where A is any algorithm for solving smooth strongly-convex ﬁ nite-sum minimization problems.

Original: The theorem below shows that Algorithm1 is ( ε, δ ) -DP with near-optimal guarantees on the excess risk.
Revised: The theorem below shows that Algorithm 1 is ( ε , δ ) -DP with near-optimal guarantees on the excess risk.

Original: A proof is provided in Appendix A.
Revised: A proof is provided in Appendix A.

Revised: Run the algorithm A on the smooth strongly-convex ﬁ nite-sum minimization problem

Revised: [Equation]

Revised: ˜

Revised: [Equation]

Revised: to obtain the output x k such that ∥ x k − ˆ x ∗ k ∥ ≤ L/ ( µ k ¯ n ) with probability at least 1 − δ / 4 , where ˆ x ∗ k = arg min x ∈ X ˆ F k ( x ) is the empirical optimal solution.

Original: Theorem 3.3.
Revised: Theorem 3.3.

Original: Under Assumption 3.2, Algorithm 1 is ( ε, δ ) -DP
Revised: Under Assumption 3.2, Algorithm 1 is ( ε , δ ) -DP

Original: and its output ˜ x satisfies(excess empirical risk)
Revised: and its output ˜ x satis ﬁ es (excess empirical risk)

Original: [Equation]
Revised: [Equation]

Original: where κ = ℓ/µ is the condition number and we assume x ∗ and ˆ x ∗ S are interior points of X . min ∈X ˆ F k ( x ) ≜ (1 / ¯ n ) (cid:80) k ¯ n i =( k − 1)¯ n +1 f ( x ; ξ i )
Revised: (excess population risk)

Original: + ( µ k / 2) ∥ x − ˜ x
Revised: [Equation]

Original: k − 1 ∥ 2 to obtain the output x k such that ∥ x k − ˆ x ∗ k ∥ ≤ L/ ( µ k ¯ n ) with probability at least 1 − δ/ 4 , where ˆ x ∗ k = arg min x ∈X ˆ F k ( is the empirical optimal solution
Revised: where κ = ℓ /µ is the condition number and we assume x ∗ and ˆ x ∗ S are interior points of X .

Original: Remark 1.
Revised: Remark 1.

Original: Many algorithms for smooth strongly-convex finite-sum minimization problems guarantee
Revised: Many algorithms for smooth strongly-convex ﬁ nite-sum minimization problems guarantee

Original: E [ ˆ F S ( A ( S )) − ˆ F S (ˆ x ∗ S )] ≤ γ with O ( T ( n, κ ) log(1 /γ )) gradient evaluations, where n is the samplesize and κ is the condition number.
Revised: E [ ˆ F S ( A ( S )) − ˆ F S (ˆ x ∗ S )] ≤ γ with O ( T ( n, κ ) log(1 / γ )) gradient evaluations, where n is the sample size and κ is the condition number.

Original: Setting 1 /γ = 32 µn 2 / ( δ 2 L 2 ) satisfies the requirements of
Revised: Setting 1 / γ = 32 µn 2 / ( δ 2 L 2 ) satis ﬁ es the requirements of

Original: A by Markov’s inequality (see Appendix A).
Revised: A by Markov’s inequality (see Appendix A).

Original: The gradient complexity of Algorithm 1 is O (( n +κ ) log( n/δ )) for SVRG [25] or SARAH [37], and O (( n + √ nκ ) log( n/δ )) for Katyusha [2].
Revised: The gradient complexity of Algorithm 1 is O (( n + κ ) log( n/ δ )) for SVRG [26] or SARAH [38], and O (( n + √ n κ ) log( n/ δ )) for Katyusha [2].

Original: As a comparison, most state-of-the-art private algorithms [43, 48, 13] for smooth strongly-convexfunctions only focus on empirical problems.
Revised: As a comparison, most state-of-the-art private algorithms [44, 50, 14] for smooth strongly-convex functions only focus on empirical problems.

Original: The linear-time algorithms in Feldman et al.
Revised: The linear-time algorithms in Feldman et al.

Original: [21] achieveoptimal population guarantees but only work for κ ≤ ˜ O ( n ) since they utilize the stability of SGD.
Revised: [22] achieve optimal population guarantees but only work for κ ≤ ˜ O ( n ) since they utilize the stability of SGD.

Original: Instead, we provide a flexible framework that includes various base methods without the necessity toshow their algorithm-specific stability.
Revised: Instead, we provide a ﬂ exible framework that includes various base methods without the necessity to show their algorithm-speci ﬁ c stability.

Original: Further, near-linear time-complexity can be attained using fastvariance reduction-based methods.
Revised: Further, near-linear time-complexity can be attained using fast variance reduction-based methods.

Original: Similar results existed in [32], but their base algorithm A onlyhas in-expectation guarantees, which brings a critical challenge to the design of private mechanisms.
Revised: Similar results existed in [33], but their base algorithm A only has in-expectation guarantees, which brings a critical challenge to the design of private mechanisms.

Original: 3.2 Near-Linear Time Algorithms for Smooth Convex Functions
Revised: 3.2 Near-Linear Time Algorithms for Smooth Convex Functions

Original: Next, we study the convex setting.
Revised: Next, we study the convex setting.

Original: A direct method is to first reduce the convex problem to a stronglyconvex one by adding a regularizer ( µ/ 2)
Revised: A direct method is to ﬁ rst reduce the convex problem to a strongly-convex one by adding a regularizer ( µ/ 2)

Original: ∥ x ∥ 2 to the objective and then apply Algorithm 1.
Revised: ∥ x ∥ 2 to the objective and then apply Algorithm 1.

Original: However,this approach only achieves a sub-optimal rate [48, 32].
Revised: However, this approach only achieves a sub-optimal rate [50, 33].

Original: Inspired by the phased-ERM [21] methodfor nonsmooth convex losses, we show that a more sophisticated multi-phase Algorithm 2 using anincreasing sequence of regularization parameters { µ k } can achievethe near-optimal guarantees onthe population loss for smooth DP-SCO.
Revised: Inspired by the phased-ERM [22] method for nonsmooth convex losses, we show that a more sophisticated multi-phase Algorithmusing an increasing sequence of regularization parameters { µ k } can achieve near-optimal guarantees on the population loss for smooth DP-SCO.

Original: The increasing regularization parameter µ k in the algorithmensures that both added DP noise and approximation errors  from the regularizer can be properlycontrolled.
Revised: The increasing regularization parameters ensure that both added DP noise and approximation errors coming from the regularizer can be properly controlled.

Original: The output of Algorithm 2 satisfies the following guarantees, as proved in Appendix A.
Revised: The output of Algorithm 2 satis ﬁ es the following guarantees, as proved in Appendix A.

Original: Theorem 3.4.
Revised: Theorem 3.4.

Original: Let Assumption 3.1 hold.
Revised: Let Assumption 3.1 hold.

Original: Suppose the optimal solution x ∗ ∈ arg min x ∈X F ( x )satisfies that ∥ x ∗ ∥ ≤ D .
Revised: Suppose there exists at least one optimal solution x ∗ ∈ arg min x ∈ X F ( x ) such that ∥ x ∗ ∥ ≤ D .

Original: Then Algorithm 2 is ( ε, δ ) -DP and its output ˜ x K satisfies
Revised: Algorithm 2 is ( ε , δ ) -DP and its output ˜ x K satis ﬁ es

Original: [Equation]
Revised: [Equation]

Original:  for the excess population risk when setting µ [Equation] L/ [Equation]
Revised: , for the excess population risk when setting µ = ( L/D ) max  1 / √ n, 14 log( n ) p d log(2 .

Original: [Equation]
Revised: 5 / δ ) / ( n ε ) .

Original: Remark 2.
Revised: Remark 2.

Original: By Remark 1 and the fact that (cid:80) Kk =1 1 /µ k ≤ O (( D/L ) √ n ) , the total gradient complexity of Algorithm 2 is O (( n + √ nℓD/L ) log( n/δ )) using SVRG [25] or SARAH [37], and
Revised: By Remark 1 and the fact that P K k =1 1 /µ k ≤ O (( D/L ) √ n ) , the total gradient complexity of Algorithm 2 is O (( n + √ n ℓ D/L ) log( n/ δ )) using SVRG [26] or SARAH [38], and

Original: O (( n + n 3 / 4 (cid:112) ℓD/L ) log( n/δ )) for Katyusha [2] (see Appendix A).
Revised: O (( n + n 3 /p ℓ D/L ) log( n/ δ )) for Katyusha [2] (see Appendix A).

Original: For the sake of simplicity, thedependence on D/L in the complexity will be left out in the following.

Original: We also point out that thesmoothness assumption here is not necessary to derive the utility bound, but allows the use of fastaccelerated algorithms for solving the regularized ERM problems.
Revised: We also point out that the smoothness assumption here is not necessary to derive the utility bound, but allows the use of fast accelerated algorithms for solving the regularized ERM problems.

Original: Bassily et al.

Original: [6] first proved optimal population guarantees for smooth convex functions, but theiralgorithm needs O ( n 3 / 2 √ ε ) gradient queries.

Original: The algorithms in Feldman et al. [21] achieve lineartime-complexity, and require that ℓ ≤ O (max {√ n, (cid:112) d log(1 /δ ) /ε } ) rooted in the stability analysisof SGD. In contrast, our framework can achieve near-optimal population guarantees using anyalgorithm without additional restrictions to the smoothness parameter.
Revised: Input: Dataset

Original: By equipping with variance

Original: Input 1: Run algorithm A on smooth strongly-convex–strongly-concave finite-sum saddle point problem miny
Revised: 1: Run algorithm A on strongly concave ﬁ nite sum problem min

Original: ˆ
Revised: Y

Original: S x
Revised: S x

Original: ) = n ) 1 f ( x, y  ξ i ) to obtain the output ( A x ( S ) , A y ( S )) such that with probability at least 1 − δ/ 8 ,
Revised: P ni =1 f ( x, y ; ξ i ) to obtain the output ( A x ( S ) , A y ( S )) such that with probability at least 1 − δ / 8 ,

Original: [Equation]
Revised: [Equation]

Original: (3) where (ˆ x ∗ S , ˆ y ∗ S ) is the saddle point of ˆ F S ( x, y ) and we let µ :
Revised: where (ˆ x ∗ S , ˆ y ∗ S ) is the saddle point of ˆ F S ( x, y ) and we let µ :

Original: = min { µ x , µ y } .
Revised: = min { µ x , µ y } .

Original: reduction-based methods, near-linear time-complexity can also be attained when ≤ O 
Revised: Bassily et al. [7] ﬁ rst proved optimal population guarantees for smooth convex functions, but their algorithm needs O ( n 3 / 2 √ ε ) gradient queries.

Original:  √ 
Revised: The algorithms in Feldman et al. [22] achieve linear time-complexity, and require that ℓ ≤ O (( L/D ) max { √ n, p d log(1 / δ ) / ε } ) rooted in the stability analysis of SGD. In contrast, our framework can achieve near-optimal population guarantees using any algorithm without additional restrictions to the smoothness parameter. By equipping with variance reduction-based methods, near-linear time-complexity can also be attained when ℓ ≤ O ( √ nL/D ) .

Original:  the best of our knowledge, this is the first time that sophisticated optimization algorithms besides
Revised: To the best of our knowledge, this is the ﬁ rst time that sophisticated optimization algorithms besides

Original: SGD are proven to obtain near-optimal population guarantees in near-linear time.
Revised: SGD are proven to obtain near-optimal population guarantees in near-linear time.

Original: Additionally, in theregime that Ω( √ n ) ≤ ℓ ≤ O ( n ) where previous smooth DP-SCO algorithms [6, 21] fail to provideoptimal guarantees, our framework still achieves a near-optimal rate with a better gradient complexity  O ( n 3 / 4 √ ℓ  )using Katyusha compared to the best nonsmooth DP-SCO algorithms [4, 29].
Revised: Additionally, in the regime that Ω ( √ nL/D ) ≤ ℓ ≤ O ( nL/D ) where previous smooth DP-SCO algorithms [7, 22] fail to provide optimal guarantees, our framework still achieves a near-optimal rate with a better gradient complexity ˜ O ( n 3 /p ℓ D/L ) compared to the state-of-the-art nonsmooth DP-SCO algorithms [4, 30].

Original:  Differentially Private Stochastic Minimax Optimization
Revised:  Differentially Private Stochastic Minimax Optimization

Original: Using the same ideas as the minimization case in the previous section, we develop differentiallyprivate algorithms for stochastic minimax optimization in (1).
Revised: Using the same ideas as the minimization case in the previous section, we develop differentially private algorithms for stochastic minimax optimization in (1).

Original: 4.1 Near-Linear Time Algorithms for Smooth Strongly-Convex–Strongly-Concave Functions
Revised: 4.1 Near-Linear Time Algorithms for Smooth Strongly-Convex–Strongly-Concave Functions

Original: First, we study the strongly-convex–strongly-concave (SC-SC) case with the following assumptions.
Revised: First, we study the strongly-convex–strongly-concave (SC-SC) case with the following assumptions.

Original: Assumption 4.1.
Revised: Assumption 4.1.

Original: For any vector ξ , f ( x, y ; ξ ) is L -Lipschitz and ℓ -smooth on the closed convexdomain X × Y ⊂ R d x × R d y .
Revised: For any vector ξ , f ( x, y ; ξ ) is L -Lipschitz and ℓ -smooth on the closed convex domain X × Y ⊂ R d x × R d y .

Original: Moreover, f ( · , y ; ξ ) is convex on X for any y ∈ Y , and f ( x, · ; ξ ) isconcave on Y for any x ∈ X .
Revised: Moreover, f ( · , y ; ξ ) is convex on X for any y ∈ Y , and f ( x, · ; ξ ) is concave on Y for any x ∈ X .

Original: Assumption 4.2.
Revised: Assumption 4.2.

Original: For any vector ξ , f ( x, y ; ξ ) satisfies Assumption 4.1 and f ( · , y ; ξ ) is µ x -stronglyconvex on X for any y ∈ Y , and f ( x, · ; ξ ) is µ y -strongly concave on Y for any x ∈ X .
Revised: For any vector ξ , f ( x, y ; ξ ) satis ﬁ es Assumption 4.1 and f ( · , y ; ξ ) is µ x -strongly convex on X for any y ∈ Y , and f ( x, · ; ξ ) is µ y -strongly concave on Y for any x ∈ X .

Original: In a manner similar to the minimization case, Zhang et al.
Revised: In a manner similar to the minimization case, Zhang et al.

Original: [49] showed that the empirical saddle pointis stable, which also implies its generalization error (see Lemma B.2 in the appendix).
Revised: [51] showed that the empirical saddle point is stable, which also implies its generalization error (see Lemma B.2 in the appendix).

Original: As a directconsequence, any algorithm A whose output is sufficiently close to the empirical saddle point hasbounded sensitivity.
Revised: As a direct consequence, any algorithm A whose output is suf ﬁ ciently close to the empirical saddle point has bounded sensitivity.

Original: This observation leads to Algorithm 3 for SC-SC DP-SMO with guaranteesgiven in the theorem below.
Revised: This observation leads to Algorithm 3 for SC-SC DP-SMO with guarantees given in the theorem below.

Original: Here A can be any method for smooth SC-SC finite-sum minimaxproblems, and smoothness allows us to obtain efficient algorithms.
Revised: Here A can be any method for smooth SC-SC ﬁ nite-sum minimax problems, and smoothness allows us to obtain ef ﬁ cient algorithms.

Original: Theorem 4.3.
Revised: Theorem 4.3.

Original: Under Assumption 4.2, let saddle points (ˆ x ∗ S , ˆ y ∗ S ) and ( x ∗ , y ∗ ) be interior points ofthe domain X × Y .
Revised: Under Assumption 4.2. Let saddle points (ˆ x ∗ S , ˆ y ∗ S ) and ( x ∗ , y ∗ ) be interior points of the domain X × Y .

Original: Then Algorithm 3 is ( ε, δ ) -DP
Revised: Then Algorithm 3 is ( ε , δ ) -DP

Original: and its output (˜ x, ˜ y ) satisfies the following utilitybounds on the empirical and population strong duality gap:
Revised: and its output (˜ x, ˜ y ) satis ﬁ es the following utility bounds on the empirical and population strong duality gap:

Original: [Equation]
Revised: [Equation]

Original: , where we let µ = min { µ x , µ y } , κ x = ℓ/µ x , κ y = ℓ/µ y , κ = ℓ/µ , and d = max { d x , d y } .
Revised: where we let µ = min { µ x , µ y } , κ x = ℓ /µ x , κ y = ℓ /µ y , κ = ℓ /µ , and d = max { d x , d y } .

Original: Remark 3.
Revised: Remark 3.

Original: Existing methods for finite-sum saddle point problems output solutions such that
Revised: Existing methods for ﬁ nite-sum saddle point problems output solutions such that

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: max
Revised: complexity O (

Original: [Equation]
Revised: [Equation]

Original: min
Revised: log(1 /

Original: [Equation]
Revised: [Equation]

Original: ∈X ˆ F S ( x, A y ( S ))] ≤ γ with complexity O

Original: ( T ( n, κ ) log(1 /γ )) .

Original: Thus setting 1 /γ = [-
-] µn 2 / ( δL 2 ) satisfies the requirements of A (see Appendix B).
Revised: Thus setting 1 / γ = [-
-]16 µn 2 / ( δ L 2 ) satis ﬁ es the requirements of A (see Appendix B).

Original: The gradientcomplexity of Algorithm 3 is O ( nκ log( n/δ )) for Extragradient [42], O (( n + κ 2 ) log( n/δ )) for
Revised: The gradient complexity of Algorithm 3 is O ( n κ log( n/ δ )) for Extragradient [43], O (( n + κ 2 ) log( n/ δ )) for

Original: SVRG/SAGA [38], O (( n + √ nκ ) log( n/δ )) for Acc-SVRG/SAGA [38] and O (( n + √ nκ x κ y +n 3 / 4 √ κ ) log( n/δ )) for AL-SVRE [33] or Catalyst-Acc-SVRG [46]. [Equation]
Revised: SVRG/SAGA [39], O (( n + √ n κ ) log( n/ δ )) for Acc-SVRG/SAGA [39] and O (( n + √ n κ x κ y + n 3 / 4 √ κ ) log( n/ δ )) for AL-SVRE [34] or Catalyst-Acc-SVRG [48].

Original: to obtain the output ( x k , y k ) such that with probability 1 − δ/ 8 ,

Original: [Equation]

Original: where (ˆ x ∗ k , ˆ y ∗ k ) is the saddle point of the regularized empirical function ˆ F k ( x, y ) .

Original: The minimax problem is equivalent to a minimization problem on x when the domain Y is restrictedto a singleton.
Revised: The minimax problem is equivalent to a minimization problem on x when the domain Y is restricted to a singleton.

Original: As a result, the lower-bound Ω(1 / ( µ x n )+ d x log(1 /δ ) / ( µ x n 2 ε 2 )) of SC DP-SCO [5]trivially holds for SC-SC DP-SMO, and our results are near-optimal w.r.t.
Revised: As a result, the lower-bound Ω (1 / ( µ x n )+ d x log(1 / δ ) / ( µ x n 2 ε 2 )) of SC DP-SCO [6] trivially holds for SC-SC DP-SMO, and our results are near-optimal w.r.t.

Original: n and ( ε, δ ) .
Revised: n and ( ε , δ ) .

Original: The flexibleframework allows the use of off-the-shelf optimization algorithms for smooth SC-SC finite-summinimax problems from a well-studied research community [46, 33] without being aware of thealgorithm-specific stability bound.
Revised: The ﬂ exible framework allows the use of off-the-shelf optimization algorithms for smooth SC-SC ﬁ nite-sum minimax problems from a well-studied research community [48, 34] without being aware of the algorithm-speci ﬁ c stability bound.

Original: Based on this framework, we can produce the first near-linear timealgorithms for smooth SC-SC DP-SMO with near-optimal guarantees on the duality gap.
Revised: Based on this framework, we can produce the ﬁ rst near-linear time algorithms for smooth SC-SC DP-SMO with near-optimal guarantees on the duality gap.

Original: Similarrates were obtained in [32] on the  primal risk through the reduction to SC DP-SCO.
Revised: Similar rates were obtained in [33] on the population primal risk through the reduction to SC DP-SCO.

Original: However, thegeneralization error on the primal function does not always apply to the original minimax problemwhen the expectation and maximization cannot be exchanged.
Revised: However, the generalization error on the primal function does not always apply to the original minimax problem when the expectation and maximization cannot be exchanged.

Original: 4.2 Near-Linear Time Algorithms for Smooth Convex-Concave Functions
Revised: 4.2 Near-Linear Time Algorithms for Smooth Convex-Concave Functions

Original: In this section, we focus on the more general case when the objective is smooth and convex-concave.
Revised: In this section, we focus on the more general case when the objective is smooth and convex-concave.

Original: We continue to build private algorithms using the stability and generalization results of empiricalsolutions.
Revised: We continue to build private algorithms using the stability and generalization results of empirical solutions.

Original: When the function is convex-concave, the following results for the regularized empiricalproblems can be applied by adding an SC-SC regularizer.
Revised: When the function is convex-concave, the following results for the regularized empirical problems can be applied by adding an SC-SC regularizer.

Original: Here, the stability bound is slightly tighterthan the one in [49] and a proof is provided in Appendix B for completeness.
Revised: Here, the stability bound is slightly tighter than the one in [51] and a proof is provided in Appendix B for completeness.

Original: Lemma 4.4.
Revised: Lemma 4.4.

Original: Consider a stochastic minimax problem such that f ( x, y ; ξ ) is convex-concave and LLipschitz with a µ x -strongly convex µ y -strongly concave regularizer G ( x, y ) .
Revised: Consider a stochastic minimax problem such that f ( x, y ; ξ ) is convex-concave and LLipschitz with a µ x -strongly convex µ y -strongly concave regularizer G ( x, y ) .

Original: Let µ = min { µ x , µ y }and denote the empirical saddle point of function ˆ F S ( x, y )
Revised: Let µ = min { µ x , µ y } and denote the empirical saddle point of function ˆ F S ( x, y )

Original: + G ( x, y ) as (ˆ x ∗ S , ˆ y ∗ S ) given dataset Swith n i.i.d.
Revised: + G ( x, y ) as (ˆ x ∗ S , ˆ y ∗ S ) given dataset S with n i.i.d.

Original: samples.
Revised: samples.

Original: Then for any neighboring datasets S ∼ S ′ , we have
Revised: Then for any neighboring datasets S ∼ S ′ , we have

Original: [Equation]
Revised: [Equation]

Original: The stability result implies the generalization error of the empirical solution can be bounded as
Revised: The stability result implies the generalization error of the empirical solution can be bounded as

Original: [Equation]
Revised: [Equation]

Original: measured by the population weak duality gap.
Revised: measured by the population weak duality gap.

Original:  In order to derive a near-optimal algorithm for convex-concave objectives, we need to carefullydesign the regularization parameters for the primalvariable x and the dualvariable y to control bothapproximation errors and DP noise.Unlike DP-SCO, the interaction between the primal x and thedual y puts additional challenges in the analysis.
Revised: Unlike DP-SCO, the interaction between the primal variable x and the dual variable y puts additional challenge in the analysis. In order to derive a near-optimal algorithm for convex-concave objectives, we need to carefully design the regularization parameters for the primal x and the dual y to control both approximation errors and DP noise.

Original: Based on the phased output perturbation framework,we propose a novel algorithm for convex-concave DP-SMO in Algorithm 4.
Revised: Based on the phased output perturbation framework, we propose a novel algorithm for convex-concave DP-SMO in Algorithm 4.

Original: The fixed parameter µ ensures the approximation error from adding regularization for the dual isproperly bounded.
Revised: The small ﬁ xed parameter µ ensures the approximation error from adding regularization for the dual is properly bounded.

Original: The role of the increasing parameter µ k is the same as Algorithm 2.
Revised: The role of the increasing parameter µ k is the same as Algorithm 2.

Original: We only add noise to the primal solutions and output ˜ x K in Algorithm 4.
Revised: We only add noise to the primal solutions and output ˜ x K in Algorithm 4.

Original: A natural way to derive a correspondingdual solution is to solve the smooth concave maximization problem max y ∈Y E [ F (˜ x K , y )] with DPconstraints.
Revised: A natural way to derive a corresponding dual solution is to solve the smooth concave maximization problem max y ∈ Y E [ F (˜ x K , y )] with DP constraints.

Original: We instead provide an alternative method that is symmetric to Algorithm 4with the sameframework by switching the role of the primal and the dual.
Revised: We instead provide an alternative method that is symmetric to Algorithm 4 by switching the role of the primal and the dual.

Original: This phased algorithm iteratively solves
Revised: This phased algorithm iteratively solves

Original: [Equation]

Original: 

Original: [Equation]
Revised: [Equation]

Original: at phase k and only perturbs the dual variables.
Revised: at phase k and only perturbs the dual variables.

Original: The detailed Algorithm 5 is provided in Appendix B.
Revised: The detailed Algorithm 5 is provided in Appendix B.

Original: Its output ˜ y K together with the output ˜ x K of Algorithm 4 satisfy ( ε, δ ) -DP by basic composition in
Revised: Its output ˜ y K together with the output ˜ x K of Algorithm 4 satisfy ( ε , δ ) -DP by basic composition in

Original: Lemma 2.1.
Revised: Lemma 2.1.

Original: The guarantees are shown in the following theorem with detailed proofs in Appendix B.
Revised: The guarantees are shown in the following theorem with detailed proofs in Appendix B.

Revised: Run the algorithm A on the smooth SC-SC ﬁ nite-sum saddle point problem

Revised: [Equation]

Revised: to obtain the output ( x k , y k ) such that with probability 1 − δ / 8 ,

Original: Theorem 4.5.
Revised: [Equation] where (ˆ x ∗ k , ˆ y ∗ k ) is the saddle point of the regularized empirical function ˆ F k ( x, y ) . [Equation] [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: Theorem 4.5.

Original: Let Assumption 4.1 hold.
Revised: Let Assumption 4.1 hold.

Original: Suppose max {∥ x ∥ , ∥ y ∥} ≤ D for all x ∈ X and y ∈ Y .
Revised: Suppose max { ∥ x ∥ , ∥ y ∥ } ≤ D for all x ∈ X and y ∈ Y .

Original: Then the composition of Algorithm 4 and 5 is ( ε, δ ) -DP
Revised: Then the composition of Algorithm 4 and 5 is ( ε , δ ) -DP

Original: and the output (˜ x K , ˜ y K ) satisfies the followingbound on the population weak duality gap:
Revised: and the output (˜ x K , ˜ y K ) satis ﬁ es the following bound on the population weak duality gap:

Original: [Equation]
Revised: [Equation]

Original: when setting µ = ( L/D ) max (cid:110) 2 / √ n, 13 log( n ) (cid:112) d log(5 /δ ) / ( nε ) (cid:111) , where d = max { d x , d y } .
Revised: when setting µ = ( L/D ) max [Equation] n, 13 log( n ) p d log(5 / δ ) / ( n ε ) o , where d = max { d x , d y } .

Original: Remark 4.
Revised: Remark 4.

Original: By Remark 3, we can set 1 /γ = 16 µ ¯ n 2 / ( δL 2 ) to satisfy the requirements of A ineach phase.
Revised: By Remark 3, we can set 1 / γ = 16 µ ¯ n 2 / ( δ L 2 ) to satisfy the requirements of A in each phase.

Original: For example, if we use Extragradient [42], the total complexity is2 (cid:80)
Revised: For example, if we use Extragradient [43], the total complexity is

Original: Kk
Revised: P

Original:  =1 O ((¯ n ( ℓ +µ k ) /µ )
Revised: Kk =1 O ((¯ n ( ℓ + µ k ) /µ )

Original: log(¯ n/δ ))
Revised: log(¯ n/ δ ))

Original: =
Revised: =

Original: O (( n 3 / 2 ℓ  + n 2 ) log( n/δ )) .
Revised: O (( n 3 / 2 ℓ D/L + n 2 ) log( n/ δ )) .

Original: After similar calculations, the complexity becomes O (( nℓ + n 5 / 4 ) log( n/δ ))
Revised: After similar calculations, the complexity becomes O (( n ℓ D/L + n 5 / 4 ) log( n/ δ ))

Original: if A is AL-SVRE [33] or Catalyst-Acc-SVRG [46] noticing that thecondition number ( ℓ + µ k ) /µ can be as large as O ( n ) .
Revised: if A is AL-SVRE [34] or Catalyst-Acc-SVRG [48] noticing that the condition number ( ℓ + µ k ) /µ can be as large as O ( n ) .

Original: Here, the dependence on D/L is omitted.

Original: Remark 5.
Revised: Remark 5.

Original: To achieve near-linear time-complexity, we utilize the special structures of our regularizedproblems in Algorithm 4 and 5.
Revised: To achieve near-linear time-complexity, we utilize the special structures of our regularized problems in Algorithm 4 and 5.

Original: (i)
Revised: (i)

Original: Objective functions of the regularized problems are separablewith different smoothness parameters w.r.t.
Revised: Objective functions of the regularized problems are separable with different smoothness parameters w.r.t.

Original: x and y .
Revised: x and y .

Original: Applying the algorithm in Jin et al.
Revised: Applying the algorithm in Jin et al.

Original: [24] that isoptimized for such special case, we can achieve the overall complexity O (( n + nℓ ) log( n/δ )) .
Revised: [25] that is optimized for such special case, we can achieve the overall complexity O (( n + n ℓ D/L ) log( n/ δ )) .

Original: (ii) Theregularization terms with potentially large smoothness parameters are prox-friendly 2 .
Revised: (ii) The regularization terms with potentially large smoothness parameters are prox-friendly 2 .

Original: SVRG/SAGAand its accelerated version [38] which leverage this property can be used, and the complexity onlydepends on the smoothness parameter of f ( x, y ; ξ ) .
Revised: SVRG/SAGA and its accelerated version [39] which leverage this property can be used, and the complexity only depends on the smoothness parameter of f ( x, y ; ξ ) .

Original: The total complexity is O (( n + nℓ 2 ) log( n/δ ))for SVRG/SAGA, and O (( n + nℓ ) log( n/δ )) for Acc-SVRG/SAGA, noticing that now the conditionnumber ℓ/µ is only O ( √ nℓ ) .
Revised: The total complexity is O (( n + n ( ℓ D/L ) 2 ) log( n/ δ )) for SVRG/SAGA, and O (( n + n ℓ D/L ) log( n/ δ )) for Acc-SVRG/SAGA, noticing that now the condition number ℓ /µ is only O ( √ n ℓ D/L ) .

Original: More details are provided in Appendix B.
Revised: More details are provided in Appendix B.

Original: Algorithm 4 can be extended to the smooth C-SC case or the SC-C case, where near-linear timenear-optimal algorithms are also provided (see Appendix B).
Revised: Algorithm 4 can be extended to the smooth C-SC case or the SC-C case, where near-linear time near-optimal algorithms are also provided (see Appendix B).

Original: According to the lower-bound discussedin [10], our utility guarantee is optimal up to logarithmic terms.
Revised: According to the lower-bound discussed in [11], our utility guarantee is optimal up to logarithmic terms.

Original: Although the primal problemmin x ∈X Φ( x ) ≜ max y ∈Y F ( x, y ) is not necessarily smooth when f ( x, y ; ξ ) is smooth and convexconcave, we provide several instances of the flexible framework that achieve near-optimal rates innear-linear time.
Revised: Although the primal problem min x ∈ X Φ ( x ) ≜ max y ∈ Y F ( x, y ) is not necessarily smooth when f ( x, y ; ξ ) is smooth and convexconcave, we provide several instances of the ﬂ exible framework that achieve near-optimal rates in near-linear time.

Original: This improves upon current results for DP-SMO [10, 47] and gives a new examplewhere near-linear time algorithms are available for nonsmooth DP-SCO.
Revised: This improves upon current results for DP-SMO [11, 49] and gives a new example where near-linear time algorithms are available for nonsmooth DP-SCO.

Revised: Our results suggest that for nonsmooth DP-SCO problems, if there exists some smooth convex-concave minimax reformulation, then near-optimal rates can be attained in near-linear time.

Revised: For example, the nonsmooth convex problem min x ∈ R dx ∥

Revised: Ax − b ∥ 1 given A ∈ R d y × d x and b ∈ R d y is equivalent to the smooth convexconcave minimax problem min x ∈ R dx max y ∈ R dy , ∥ y ∥ ∞ ≤ 1 y ⊤ ( Ax − b ) .

Original: It remains an open questionwhether optimal utility bound can be obtained in linear time for general nonsmooth problems.
Revised: However, it remains an open question whether optimal utility bound can be obtained in linear time for general nonsmooth problems.

Original:  Conclusion
Revised:  Conclusion

Original: We provide a general framework for both smooth DP-SCO and DP-SMO problems with near-optimalprivacy-utility trade-offs.
Revised: We provide a general framework for both smooth DP-SCO and DP-SMO problems with the nearoptimal privacy-utility trade-off.

Original: The flexible framework allows to bring various off-the-shelf, fast convergentnon-private optimization algorithms into the DP domain.
Revised: The ﬂ exible framework allows to bring various off-the-shelf, fast convergent non-private optimization algorithms into the DP domain.

Original: Using the framework, we enrich the class ofnear-linear time algorithms for smooth DP-SCO and provide the first near-linear time algorithms forsmooth DP-SMO.
Revised: Using the framework, we enrich the class of near-linear time algorithms for smooth DP-SCO and provide the ﬁ rst near-linear time algorithms for smooth DP-SMO.

Original: For future work, it is interesting to study whether the logarithmic terms in gradientcomplexity and final utility bound of our algorithms can be further removed.
Revised: For future work, it is interesting to study whether the logarithmic terms in gradient complexity and the ﬁ nal utility bound of our algorithms can be further removed.

Original: Another direction is toderive simpler private algorithms for smooth convex-concave DP-SMO based on the special structureof the regularized problems.
Revised: Another direction is to derive simpler private algorithms for smooth convex-concave DP-SMO based on the special structure of the regularized problems.

Original: Moreover, we believe our framework also opens the door to simplerderivatives with possibly improved complexities for nonsmooth DP-SCO, nonsmooth DP-SMO, andeven more general forms of optimization problems such as variational inequalities and games.
Revised: Moreover, we believe our framework also opens the door to simpler derivatives with possibly improved complexities for nonsmooth DP-SCO, nonsmooth DP-SMO, and even more general forms of optimization problems such as variational inequalities and games.

Original: Wewill leave these for future investigation.
Revised: We will leave these for future investigation.

Original: Checklist
Revised: Acknowledgments and Disclosure of Funding

Revised: We would like to thank anonymous reviewers for the insightful feedback. L.Z. gratefully acknowledges funding by the Max Planck ETH Center for Learning Systems (CLS). This work does not relate to the current position of K.T. at Amazon.

Original: For all authors...
Revised: S.O. is supported in part by NSF grants CNS-2002664, IIS-1929955, and CCF-2019844 as a part of NSF Institute for Foundations of Machine Learning (IFML). N.H. is supported by ETH research grant funded through ETH Zurich Foundations and NCCR Automation funded through Swiss National Science Foundations; part of the work was done while N.H. was visiting the Simons Institute for the Theory of Computing.

Original:  (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately re ﬂ ect the paper’s contributions and scope?

Original: [Yes] See Section 3 for contributions in DP-SCO and Section4 for contributions in DP-SMO.
Revised: [Yes] See Section 3 for contributions in DP-SCO and Section 4 for contributions in DP-SMO.

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] See Section 5.(c)
Revised: [Yes] See Section 5.

Original:  Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A]
Revised: [N/A]

Original: This paper is purely theoretical and has no immediate societal impacts.
Revised: This paper is purely theoretical and has no immediate societal impacts.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] See Assumption3.1 and 3.2 for DP-SCO and Assumption 4.1 and 4.2 for DP-SMO.
Revised: [Yes] See Assumption 3.1 and 3.2 for DP-SCO and Assumption 4.1 and 4.2 for DP-SMO.

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] See Appendix A forDP-SCO and Appendix B for DP-SMO.
Revised: [Yes] See Appendix A for DP-SCO and Appendix B for DP-SMO.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [N/A] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
Revised: [N/A] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [N/A] (c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?
Revised: [N/A] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: [N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [N/A] 
Revised: [N/A] 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [N/A] (b) Did you mention the license of the assets?
Revised: [N/A] (b) Did you mention the license of the assets?

Original: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [N/A] (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: [N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: [N/A] (e) Did you discuss whether the data you are using/curating contains personally identi ﬁ able information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?[N/A
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: [Table]

Original:  (b) Did you describe any potential participant risks, with links to Institutional Review[Table] Board (IRB) approvals, if applicable?[N/A]
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: [Table] (c) Did you include the estimated hourly wage paid to participants and the total amount[Table] spent on participant compensation?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: [Table]