Reproducibility in Optimization: Theoretical Framework and Limits
Abstract
We initiate a formal study of reproducibility in optimization.
We deﬁne a quantitative measure of reproducibility of optimization procedures in the face of noisy or error-prone operations such as inexact or stochastic gradient computations or inexact initialization.
We then analyze several convex optimization settings of interest such as smooth, non-smooth, and strongly-convex objective functions and establish tight bounds on the limits of reproducibility in each setting.
Our analysis reveals a fundamental trade-off between computation and reproducibility: more computation is necessary (and sufﬁcient) for better reproducibility.
Introduction
Machine learned models are increasingly entering wider ranges of domains in our lives, driving a constantly increasing number of important systems.
Large scale systems can be trained in highly parallel and distributed training environments, with a large amount of randomness in training the models.
While some systems may tolerate such randomness leading to models that differ from one another every time a model retrains, for many applications, reproducible models are required, where slight changes in training do not lead to drastic differences in the model learned.
Beyond practical deployments of machine learned models, the reproducibility crisis in the machine learning academic world has also been well-documented: see [Pineau et al., 2021] and the references therein for an excellent discussion of the reasons for irreproducibility (insufﬁcient exploration of hyperparameters and experimental setups, lack of sufﬁcient documentation, inaccessible code, and different computational hardware) and for mitigation recommendations.
Recent papers [Chen et al., 2020, D’Amour et al., 2020, Dusenberry et al., 2020, Snapp and Shamir, 2021, Summers and Dinneen, 2021, Yu et al., 2021] have also demonstrated that even when models are trained on identical datasets with identical optimization algorithms, architectures, and hyperparameters, they can produce signiﬁcantly different predictions on the same example.
This type of irreproducibility may be caused by multiple factors [D’Amour et al., 2020, Fort et al., 2020, Frankle et al., 2020,
Shallue et al., 2018, Snapp and Shamir, 2021, Summers and Dinneen, 2021], such as non-convexity of the objective, random initialization, nondeterminism in training such as data shufﬂing, parallelism, random schedules, hardware used, and round off quantization errors.
Perhaps surprisingly, even if we control for the randomness by using the same “seed" for model initialization, other factors such as numerical errors introduced due to nondeterminism of modern GPUs (see, e.g., [Zhuang et al., 2021]) may still lead to signiﬁcant differences.
It was empirically shown (see, e.g., Achille et al.
[2017]) that slight deviations early in training can lead to different optima, with substantial differences in resulting models.
Thus we are forced to accept some fundamental level of irreproducibility that persists even after ﬁxing all aspects of the training process under our control.
The goal of this paper is to initiate a formal study of the fundamental limits of irreproducibility.
In particular, we focus on the most basic training process: convex optimization .
At ﬁrst glance, it might seem surprising that convex optimization procedures can exhibit irreproducibility since they’re guaranteed to converge to an optimal solution.
However, in practice, the default convex optimization algorithms are iterative ﬁrst-order methods; methods that only use a ﬁrst-order oracle to provide an approximate gradient of the function at a given point, and converge to an approximately optimal solution.
The ﬁrst-order oracle is a source of irreproducibility.
In stochastic gradient descent, it returns a random vector whose expectation is the true gradient.
The randomness in the stochastic gradients can lead to different outcomes of the optimization process.
Similarly, there are numerical errors that can arise in the computation of the gradients due to inherent nondetermism in modern GPUs.
Beyond the ﬁrst-order oracle, irreproducibility may also arise in convex optimization procedures because the initial point is chosen randomly.
Thus, we attempt to answer the following questions for convex optimization procedures operating with the above sources of irreproducibility:
⌅ What are the fundamental limits of reproducibility for any convex optimization procedure?
⌅ Can we design practical and efﬁcient ﬁrst-order methods that achieve these limits?
We study these questions in a variety of settings; including general non-smooth convex functions, smooth convex functions, strongly-convex functions, ﬁnite-sum functions, and stochastic convex optimization, under the different sources of irreproducibility mentioned above.
To the best of our knowledge, no prior theoretical work considered such questions.
The primary contribution of this paper is conceptual: the development of a rigorous theoretical framework to study the fundamental limits of reproducibility in convex optimization.
The concepts developed in this framework can be extended easily to other settings of interest such as non-convex optimization.
The technical contribution of this paper is the development of lower bounds on the amount of reproducibility, and matching upper bounds via analysis of speciﬁc ﬁrst-order algorithms in all the different settings of convex optimization described above.
Detailed technical descriptions of the results appear below in Subsection 1.1.
At a high level, our study provides the same message for all the different optimization settings we consider.
On the lower bound side, we ﬁnd that any ﬁrst-order method would need to trade-off convergence rate (computational complexity) for more reproduciblity.
On the upper bound side, we ﬁnd that various forms of gradient descent, when run with lower step-size (and correspondingly, more iterations) already achieve the fundamental limits of reproducibility.
One (somewhat surprising) consequence of our results is that advanced techniques like regularization, variance reduction, and acceleration do not improve reproducibility over standard gradient descent methods.
We show, for example, that when optimizing a Lipschitz non-smooth convex function f on a bounded domain using a ﬁrst-order oracle that computes gradients with even vanishingly small error, two runs of any ﬁrst-order method that obtains an " suboptimal solution of f after T iterations can generate solutions that are ⌦ ( 1 / ( p T " ) ) apart in ` 2 distance.
Thus, if we run the method for the standard
T =
O ( 1 / " 2 ) iterations required to obtain " -approximate solution for a non-smooth function, then obtained solutions can deviate by ⌦ (1) distance; i.e., the method is maximally irreproducible.
To ensure that irreproducibility is small, i.e. that the solutions are within a small distance  of each other, we will need to run at least ⌦ ( 1 / ( " 2  2 ) ) iterations.
Interestingly, standard gradient descent with appropriately chosen learning rate and number of iterations already achieves this trade-off.
Our results demonstrate the challenge of reproducibility even for standard convex optimization.
While we provide matching lower and upper bounds in certain general settings, in Section 7, we outline several important open directions.
Solutions to these problems should enable better understanding of reproducibility even for deep learning.
1.1 Summary of results
Table 1 summarizes our key results for our measure of irreproducibility,
( " ,  ) -deviation
(see Deﬁnition 3).
The ( " ,  ) -deviation measures the amount of change between the outputs of two independent runs of an optimization algorithm, that is guaranteed to achieve " -suboptimality after T iterations,
[Table]
Theorem
Smooth
[Equation]
Smooth Strongly-Cvx
[Equation]
[Table]
[Equation]
[Table]
[Equation]
when the computations of the algorithm incur errors of magnitude up to  .
We speciﬁcally focus on three different sources of errors: i) stochastic gradient oracles, ii) gradient oracles with non-deterministic numerical errors (Deﬁnition 2), and iii) inexact initialization for the optimizer (Deﬁnition 1).
We analyze the deviation under these sources of errors for four types of function classes: smooth convex functions, non-smooth but Lipschitz convex functions and strongly-convex restrictions of the two.
Throughout the paper, a ^ b denotes the minimum between a and b .
All lower bounds are for ﬁrst-order iterative algorithms ( à la Nesterov [2018]) that we formally deﬁne in ( FOI ).
This is a large class of iterative optimization methods, including Stochastic Gradient
Descent (SGD), which construct successive iterates adaptively in the linear span of previous iterates.
Additionally, for smooth costs and stochastic inexact gradient oracle, we have an information theoretic lower bound of ⌦ (  2 / ( T " 2 ) ) when " .
2 (Theorem 6).
We believe such informtation-theoretic lower bounds can be shown for all the settings in this paper.
As for the upper bounds, they are all obtained using slowed-down SGD: i.e. SGD using smaller learning rates and more iterations.
For the non-strongly convex cases, one may expect to have high irreproducibility if the minima form a large ﬂat region; however, surprisingly, our upper bounds show that we can always bound the extent of irreproducibility via slowed-down SGD.
In the non-smooth cases, the main observation is that the deviation does not depend on scale of perturbation by the gradient oracle, i.e., any  > 0 can lead to fairly irreproducible solutions.
The non-stochastic gradient oracle setting is strictly harder than the stochastic setting.
Naturally, the lower and upper bounds on reproducibility are worse.
Interestingly, even though strong convexity implies uniqueness of the global optimum, which intuitively should lead to highly reproducible solutions, we show that when faced with sources of error in computations, the deviation can still be signiﬁcantly large for any algorithm.
Finally, we study reproducibility of optimization in machine learning settings.
Here we have additional structure such as ﬁnite-sum minimization (for optimizing training loss) and stochastic convex optimization (for optimizing population loss).
We deﬁne appropriate notions of errors for these problems and analyze two settings of particular interest.
Our main results (Theorem 4 and
Theorem 5) show that despite the additional structure in these problems, the bounds given by Tablefor the speciﬁc settings are nonetheless tight .
One consequence is that more sophisticated techniques for these problems such as variance reduction don’t improve reproducibility.
1.2 Related work
Related notions.
In the scientiﬁc world, the terms reproducibility and replicability are often used interchangeably, but here we distinguish the two, following Pineau et al.
Reproducibility refers to the requirement that results obtained by a computational procedure (e.g. an experiment or a statistical analysis of a data set) should be the same (or largely similar) when the procedure is repeated using the same code on the same data, whereas replicability is a different notion that requires that results be reliably the same or similar when the data are changed.
The ﬁeld of statistical hypothesis testing [Lehmann and Romano, 2005] provides rigorous and principled techniques to minimize false discoveries and thereby promote replicability.
The notion of algorithmic stability can also be seen as quantifying the amount of change in the output when a single data sample is changed.
This notion has been extensively studied in the context of providing algorithm-dependent generalization bounds [Bousquet and Elisseeff, 2001, Kutin and Niyogi, 2002] and in developing differentially private algorithms [Dwork et al., 2006, McSherry and Talwar, 2007].
In very recent concurrent work,
Impagliazzo et al.
[2022] deﬁne a notion of replicability in learning that is quite different from ours: they aim to develop algorithms that generate the exact same output with reasonable probability given a fresh sample.
Note that despite the title of their paper, technically the notion studied is replicability, not reproducibility, since they study the output of algorithms when the input data are changed.
In this paper, we speciﬁcally focus on reproducibility: how much can the results of a computation differ when it is re-run on the same data with the same code?
Hence, both hypothesis testing and algorithmic stability are orthogonal to the study in this paper, although some of our upper bounds use similar analysis techniques as algorithmic stability.
On a different note, similar to replicability, the boundary between the notions of irreproducibility and uncertainty in deep models is rather blurred.
Several papers considered different aspects of uncertainty (see, e.g., [Lakshminarayanan et al., 2017] and references therein), but this line of work has been empirical in nature.
Inexact oracles in optimization.
The optimization community has studied the consequences of using inexact or error-prone gradient oracles in optimization.
Several papers (e.g. [Aybat et al., 2020, Devolder et al., 2014, d’Aspremont, 2008, Cohen et al., 2018] have developed bounds on the optimization error incurred due to the use of inexact oracles.
While the sources of errors are similar to the ones studied in this paper, the quantities of interest in these papers are convergence rate and optimization error rather than reproducibility.
Interestingly, despite the different objective, some of the high-level conclusions are similar to our paper: for example, accelerated gradient methods do not outperform standard classical methods when used with inexact gradient oracles.
Techniques to improve reproducibility in practice.
Several recent empirical papers considered methods that can reduce levels of irreproducibility in deep models despite nondeterminism in training.
Smooth activations [Du, 2019, Mhaskar, 1997] have been shown [Shamir et al., 2020] to improve reproducibilty over popular activations, as the Rectiﬁed Linear Unit [Nair and Hinton, 2010].
Ensembles [Dietterich, 2000] leverage diversity of multiple different solutions to produce an average more reproducible one [Allen-Zhu and Li, 2020].
Co-distillation
[Anil et al., 2018] and
Anti-distillation [Shamir and Coviello, 2020] leverage ensembles to further push deployed models to be more reproducible.
Imposing constraints [Bhojanapalli et al., 2021, Shamir, 2018] forces models to prefer some solutions over others, but may come at the cost of reducing model accuracy performance.
Robustness of dynamical systems.
The upper bound results in our paper can be interpreted as robustness results of the (sub)gradient descent dynamics against disturbances.
In particular, our upper bounds can be viewed as some variants of the input-to-state stability [Sontag and Wang, 1995] results for the dynamics (see, e.g., [Tu et al., 2022, Deﬁnition 3.2]).
Problem Formulation
In this section, we deﬁne a quantitative measure of irreproducibility amenable to a theoretical analysis.
Intuitively, a computation is reproducible if it generates the exact same output given the same inputs on two different runs.
Irreproducibility arises because low-level operations of a computation produce different answers on two runs due to either randomness or non-determinism .
Our computation of interest is convex optimization via ﬁrst-order methods, where initialization and gradient computations are the primary operations that constitute the computation and are subject to errors leading to inexact outputs.
A natural measure of irreproducibility is the amount of change in the computed solution to the convex optimization problem under inexact gradient computations or inexact initialization.
However, there are two nuances that must be carefully handled here.
First, a trivial procedure which ignores its input and outputs a constant solution is perfectly reproducible!
Unfortunately, it is perfectly useless as a convex optimization procedure as well.
Thus, in order to compare different procedures by their reproducibility metrics, we must assume that the procedures are guaranteed to converge to an optimal solution.
The second nuance is that we need to assume that the errors in the gradient or initialization computations are bounded in some manner.
Evidently, without such an assumption, any non-trivial convex optimization procedure will be extremely irreproducible.
We now use the above considerations to develop a precise deﬁnition of a measure of irreproducibility.
Convex function classes.
We assume that the function to be optimized is chosen from a certain class, F , of convex functions, along with their domains, satisfying suitable regularity conditions (e.g. Lipschitzness, smoothness, strong-convexity, etc.) to develop convergence rates.
For clarity, we will suppress exact dependence on smoothness, Lipschitzness, and strong-convexity parameters.
In particular, “smooth” will denote a convex function whose gradients are O (1)
-Lipschitz continuous, “non-smooth” a convex function that is O (1) -Lipschitz continuous, and “strongly-convex” an ⌦ (1) - strongly-convex function.
Here, O (1) and ⌦ (1) denote universal constants independent of the dimension or other problem dependent quantities, which we leave unspeciﬁed to ease the exposition.
Convex optimization procedures.
A ﬁrst-order convex optimization procedure for F is an algorithm that, given any function f 2 F , and access to two (potentially noisy) oracles – an initialization oracle, which generates the initial point, and a gradient oracle, which computes gradients for f at any given query point – generates a candidate solution x out for the problem of minimizing f over its domain.
Note that the algorithm can only access f via the oracles provided.
We call such an algorithm " -accurate if it guarantees that E f ( x out )  inf x 2 dom f f ( x )  " , where the expectation is over any randomness in the computation of x out .
Several of our lower bounds require more structure for the algorithm: speciﬁcally, a ﬁrst-order iterative (FOI) algorithm ( à la Nesterov [2018]) is one that starting from the point x 0 generated by the initialization oracle, constructs successive iterates x t =
[Equation] [Equation]
for some
[Equation]
where g ( x i ) is the output of the gradient oracle query at x i , and outputs x T for some integer T > 0 .
We emphasize that for all t , the coefﬁcients  ( t ) i can be chosen adaptively based on all the previous computations.
The above class of algorithms is a canonical one to consider when proving lower bounds against gradient oracle based optimization algorithms.
We refer readers to [Nesterov, 2018, §2.1.2] for more background.
For the case of nonsmooth costs, we additionally assume that the coefﬁcient of the latest gradient is nonzero, i.e.,  ( t ) t  1 6 = 0 for all t .
We also note that one of our lower bound results (Theorem 6) is information-theoretic (in the sense of Nemirovski and Yudin [1983]).
Sources of errors in computation.
Errors arise due to inexactness in the outputs of the initialization or gradient oracles.
Queries to these oracles on two different runs of the same algorithm might yield different outputs, but we will control the errors by assuming that the outputs are close to some reference point (that remains ﬁxed over different runs) in a suitable metric.
Deﬁnition 1 (  -bounded inexact initialization oracle) .
Given a function f 2 F and a reference initialization point x ref 0 2 dom f , a  -bounded inexact initialization oracle for f is one that generates an initial point x 0 2 dom f such that k x 0  x ref 0 k   .
The gradient computation oracle is said to be  -bounded if for any f 2 F and any point x 2 dom f , it outputs a vector g ( x ) such that E k g ( x )  r f ( x ) k 2   2 for some r f ( x ) 2 @ f
( x ) , where the expectation is over any randomness in the computation of g ( x ) .
We consider both stochastic and non-stochastic inexact  -bounded gradient oracles.
A stochastic gradient oracle has the additional property that its output g ( x ) is a random vector such that E g ( x ) =
r f ( x ) , with different queries being independent of each other.
Stochastic inexact gradient oracles arise naturally in machine learning applications due to randomness in minibatching.
Non-stochastic inexact gradient oracles model non-deterministic numerical errors due to the accumulation of ﬂoating point errors; for giant machine learning models with billions of parameters, individual ﬂoating point errors could add up to a noticeable large error.
We formally deﬁne the two types of inexact oracles below.
Deﬁnition 2 (  -bounded inexact gradient oracle) .
Given a function f 2 F , and x 2 dom f , let @
f
( x ) denote the
sub
-differential of f at x .
(a) A stochastic inexact  -bounded gradient oracle outputs a random vector g ( x ) such that E g ( x ) =
r f ( x ) and E k g ( x )  r f ( x ) k 2   2 for some r f ( x ) 2 @
f
( x ) .
The expectation is over the randomness in g ( x ) which is also assumed to be independent for each oracle call.
(b) A non-stochastic inexact  -bounded gradient oracle outputs a non-deterministic vector g ( x ) such that k g ( x )  r f ( x ) k 2   2 for some r f ( x ) 2 @ f ( x ) .
Measure of irreproducibility.
Let A be a ﬁrst-order, " -accurate convex optimization procedure for F with access to either a  -bounded initialization oracle (and an exact gradient oracle), or a  -bounded gradient oracle (and an exact initialization oracle).
The ( " ,  ) -deviation is
Deﬁnition 3 ( ( " ,  ) -deviation) .
Given a function f 2 F , let x f and x 0 f denote the outputs of A on two independent runs of A that result in " -accurate solutions.
(a)
If a stochastic inexact  -bounded gradient oracle is used, the ( " ,  ) -deviation of A is deﬁned as sup f 2 F E k x f  x 0 f k 2 where the randomness is over the stochastic oracle in the two runs.
(b) If either an inexact  -bounded initialization oracle or a  -bounded non-stochastic inexact gradient oracle is used, the ( " ,  ) -deviation of A is deﬁned as sup f 2 F sup k x f  x 0 f k 2 , where the inner supremum is over the two runs.
Here we note that x f is not necessarily the last iterate of a ﬁrst-order algorithm.
Depending on cases, sometimes we choose x f to be the (weighted) average iterate.
We consider k x f  x 0 f k 2 for the notion of deviation instead of k x f  x ⇤ k 2 ; note that x ⇤ may not even be unique without strong convexity.
Other notions?
Alternate deﬁnitions are certainly plausible.
For example, in machine learning applications, the computed solution x may be used as a parameter vector for a predictor function g x which maps inputs z to real-valued predictions ˆ y whose quality is measured by a loss function ` (ˆ y, y ) where y is the true label of z .
Let x and x 0 be outputs of two different runs of the optimization algorithm.
Then one can also deﬁne ( " ,  ) -deviation based on prediction reproducibility : e.g. sup z | g x ( z )  g x 0 ( z ) | or E z | g x ( z )  g x 0 ( z ) | , or loss reproducibility : e.g. sup ( z,y ) | ` ( g x ( z ) , y )  ` ( g x 0 ( z ) , y )
| or E ( z,y )
| ` ( g x ( z ) , y )  ` ( g x 0 ( z ) , y ) | , where the expectation is over the distribution of the examples.
Nevertheless, we adopt Deﬁnition 3, which is based on parameter reproducibility , in this paper for the following reasons:
• The convex optimization problems studied in this paper are more basic/fundamental than more structured ML optimization.
To the best of our knowledge, there is no pre-existing theory even for this basic setting.
Parameter reproducibility is a more natural deﬁnition here since there is no notion of prediction or loss.
In many ML applications, the predictor function g x is Lipschitz in x for any input z .
In such cases, ( " ,  ) -deviation bounds for parameter reproducibility immediately transform into ( " ,  ) - deviation bounds for prediction reproducibility.
Similar transformations are also generally possible ( " ,  ) -deviation bounds for loss reproducibility.
• Without knowledge of how a learned parametric function is deployed for making predictions, it is difﬁcult to analyze prediction reproducibility even in ML settings.
Hence, parameter deviations provide a reasonable ﬁrst approximation.
Furthermore, in real systems, we often optimize multiple metrics (e
g. performance on sub-segments of populations, for fairness).
Parameter reproducibility gives greater assurance on all metrics.
A similar argument applies when the test distribution is different from the training distribution.
Finally, several recent works [Shamir et al., 2020, D’Amour et al., 2020] have empirically observed that even if two different runs of the algorithm resulted in parameters that have nearly the same loss, the predictions on test examples could be very different.
One reason this happens is because surrogate losses are used in place of the true metric for optimization.
So simply achieving loss reproducibility may not be sufﬁcient for practical applications.
Reproducibility with Stochastic Inexact Gradient Oracles
In this section we consider reproducibility of optimizing a convex function f 2 F where we can access f only via a stochastic gradient oracle (see Deﬁnition 2).
This setting covers several important
ML optimization scenarios, e.g., when the training data is randomly sampled from a population or when the selection of mini-batches is randomized.
Our main result is the following theorem.
Theorem 1.
For any " ,  > 0 , and number of iterations T , the ( " ,  ) -deviation for optimizing convex functions with a stochastic inexact gradient oracle is as follows.
Unless indicated otherwise, the lower bounds hold for any FOI algorithm, and the upper bound is achieved by stochastic gradient descent for T =
⌦ ( 1 / " 2 ) in the non-strongly-convex settings and T = ⌦ ( 1 / " ) in the strongly-convex
settings.
⌅ Smooth functions : ( " ,  ) -deviation is ⇥ (  2 / ( T " 2 ) ) .
Furthermore, for the case " .
2 there is a matching information theoretic lower bound .
⌅ Smooth and strongly convex functions : ( " ,  ) -deviation is ⇥ (  2 / T ^ " ) .
⌅ Lipschitz (non-smooth) functions : ( " ,  ) -deviation is ⇥ ( 1 / ( T " 2 ) ) .
⌅ Lipschitz (non-smooth) and strongly convex functions : ( " ,  ) -deviation is ⇥ ( 1 / T ^ " ) .
This theorem is proved in several pieces, one corresponding to each setting (i.e., smooth, smooth & strongly convex etc.) and type of bound (i.e., upper or lower).
The precise details are given in
Appendix A. A few remarks are in order.
Smoothness : Intuitively, smoothness would ensure that a slight error in gradient computation need not imply catastrophic deviation in the iterates.
Our matching upper and lower bounds conﬁrm this intuition.
Non-smoothness : Our results show that even a slight amount of noise in the gradients can lead to drastic irreproducibility in the non-smooth case.
Intuitively, the reason for this phenomenon is that non-smooth functions have non-differentiable points where the slightest amount of noise in the gradients can lead to drastically different behavior.
This is in line with the empirical observations of
Shamir et al.
Strong convexity : Note that the deviation is smaller than other cases due to the existence of a unique minimizer; the " -accuracy in the cost already implies an O ( " ) upper bound on the deviation.
Furthermore, in all the settings, our results show that using an algorithm with a larger number of iterations is more helpful; intuitively that is because more gradient samples from the oracle can help reduce the sample noise as an averaging.
For example, for smooth functions, if a gradient descent type method is run for the standard T
=
O
( 1 / " 2 ) iterations, then the solution obtained might still suffer deviation of  2 , which is independent of " .
Thus, in order to obtain low deviation, we are forced to run the method for !
( 1 / " 2 ) iterations.
We also note that by relying on the standard convergence rate lower bounds [Nemirovski and Yudin, 1983], the requirement on T e.g., T = ⌦ (1 / " 2 ) for the non-strongly convex setting, is without loss of generality.
Finally, we remark that the lower bound for smooth functions is information theoretic (i.e., holds against any algorithm) for " .
2 .
Reproducibility with Non-Stochastic Inexact Gradient Oracles
In this section, we study reproducibility for optimizing a function f 2 F with non-stochastic inexact gradient oracle access (Deﬁnition 2).
In particular, we establish lower and upper bounds for ﬁrstorder algorithms ( FOI ) on the ( " ,  ) -deviation (Deﬁnition 3) despite initialization with the same point x 0 2 dom f .
Recall that this setting allows us to capture reproducibility challenges due to non-deterministic numerical errors introduced by a computing device (like GPUs) during ﬂoating point computations.
The main high-level message here is that unlike in the stochastic gradient oracle case, in the non-stochastic gradient oracle setting the iteration complexity T has little to no effect on the ( " ,  ) -deviation: intuitively this is because unlike the stochastic setting, it is not possible to reduce the error in the gradients by taking more samples and averaging.
Theorem 2.
For D
= O (1) , " ,  > 0 such that   " / (2 D ) , and number of iterations T , the ( " ,  ) - deviation for optimizing convex functions whose optimum has norm at most D with a non-stochastic inexact gradient oracle is as follows.
Unless indicated otherwise, the lower bounds hold for any
FOI algorithm, and the upper bound is achieved by projected gradient descent for T =
⌦ ( 1 / " ) in the smooth settings and non-smooth strongly convex setting, and T = ⌦
( 1 / " 2 ) in the non-smooth setting.
⌅ Smooth
functions : ( " ,  ) -deviation is ⇥ (  2 / " 2 ) .
⌅ Smooth and strongly convex functions : ( " ,  ) -deviation is ⇥ (  2 ^ " ) .
⌅ Lipschitz (non-smooth) functions : ( " ,  ) -deviation is ⇥ ( 1 / ( T " 2 ) +  2 / " 2 ) .
⌅ Lipschitz (non-smooth) and strongly convex functions : ( " ,  ) -deviation is ⇥ (( 1 / T +  2 ) ^ " ) .
Like Theorem 1, this theorem is proved in several pieces; see Appendix A. Some remarks follow:
Acceleration v.s. reproducibility?
Note that for smooth functions, accelerated methods can get " -suboptimality in only T
= O ( 1 / p " ) iterations.
A natural question is if we can achieve similar ( " ,  ) -deviation for such accelerated methods.
However, it is known that accelerated methods are unstable (see e.g., [Devolder et al., 2014, Attia and Koren, 2021]), and cannot even achieve the desired " -accuracy under the inexact oracle model.
Hence, we conjecture that iteration complexity of ⌦ (1 / " ) is necessary to achieve the desired reproducibility.
Nonsmoothness: Note that the lower bound shows that if gradient descent style methods are run for the standard T = 1 / " 2 iterations, then the solutions can be forced to have ⌦ (1) deviation.
Hence, to ensure reproducibility, any method would need to have a slower convergence rate than T = 1 / " 2 .
Reproducibility with Inexact Initialization Oracles
We now study reproducibility for optimizing a function f 2 F with an inexact initialization oracle (Deﬁnition 1) and establish tight bounds on the ( " ,  ) -deviation for FOI algorithms.
An inexact initialization oracle models situations where the initial values of the parameters are set randomly, or incur some non-deterministic numerical error.
Theorem 3.
For any " ,  > 0 and number of iterations T , the ( " ,  ) -deviation for optimizing convex functions with an inexact initialization oracle is as follows.
In all cases, the lower bounds hold for any FOI algorithm, and the upper bound is achieved by gradient descent for T = ⌦ ( 1 / " ) in the smooth settings and non-smooth strongly convex setting, and T = ⌦ ( 1 / " 2 ) in the non-smooth setting.
⌅ Smooth functions : ( " ,  ) -deviation is ⇥ (  2 ) .
⌅ Smooth and strongly convex functions : ( " ,  ) -deviation is ⇥ ((exp(  ⌦ ( T ))  2 ^ " ) .
⌅ Lipschitz (non-smooth) functions : ( " ,  ) -deviation is ⇥ ( 1 / ( T " 2 ) +  2 ) .
⌅ Lipschitz (non-smooth) and strongly convex functions : ( " ,  ) -deviation is ⇥ ( 1 / T ^ " ) .
Like Theorem 1, this theorem is proved in several pieces; see Appendix A. A remark follows:
Nonsmoothness: As in the case of non-smooth optimization with inexact gradient oracles, here as well we see that even the slightest amount of inexactness in initialization can lead to non-negligible irreproducibility.
Intuitively this is a consequence of non-smoothness: If the function is not differentiable at the reference point, even a slight bit of inexactness can lead to drastically different trajectories right from the beginning.
Reproducibility in Optimization for Machine Learning
So far, we have studied reproducibility in general convex optimization with different sources of perturbation.
In machine learning, however, optimization problems come with more structure, and hence a more nuanced analysis of reproducibility is called for.
In this section, we study reproducibility in two speciﬁc optimization settings of interest in machine learning: ﬁnite sum minimization which corresponds to minimizing training loss, and stochastic convex optimization (SCO), which corresponds to minimizing population (or test) loss.
Instead of a detailed study of different types of convex functions as done in the previous sections, here we focus on a few speciﬁc cases that yield particularly interesting insights.
6.1 Optimizing Training Loss (Finite Sum Minimization)
Optimizing the training loss in ML can be cast as minimizing a function that can be written as a ﬁnite sum of component functions: f ( x ) :
= 1 m
P mi =
f
i ( x ) .
Typical optimization methods such as stochastic gradient descent can be implemented to solve such problems by iteratively sampling one of the component functions and taking its gradient.
So randomness in sampling as well as nondeterministic numerical errors in computing of gradient can lead to irreproducibility.
Randomness in sampling is a property of the algorithm and hence under the control of the algorithm designer; however non-deterministic numerical errors in gradient computations are beyond the control of the designer, and hence we aim to quantify irreproducibility caused by this speciﬁc source (numerical errors).
To capture this intuition, we consider the following inexact gradient oracle.
Deﬁnition 4 (  -bounded inexact component gradient oracle) .
A  -bounded inexact component gradient oracle takes as input i 2 { 1 , 2 , . . .
, m } and a point x , and outputs a vector
g i ( x ) such that [Equation] some
[Equation]
The number of calls to the component gradient oracle is now our primary measure of complexity.
Then, a natural question that arises is whether we can we tightly characterize reproduciblity of typical ﬁrst-order methods when given access to an inexact component gradient oracle.
We ﬁrst observe that in the special case that all f i are identical, the inexact component gradient oracle reduces to the non-stochastic inexact oracle (Deﬁnition 2).
Hence, for the smooth and non-smooth function settings, we immediately obtain lower bounds on the ( " ,  ) -deviation via our previous analysis; see Theoremand Theorem 14, respectively.
The non-smooth case in particular is interesting, as the deviation lower bound is ⌦ ( 1 / ( T " 2 )
+  2 / " 2 ) , where T is the number of oracle calls.
Now, if we use projected (full-batch) gradient descent (as used for the matching upper bound in Theorem 27), then each iteration would require m oracle calls, hence the ( " ,  ) -deviation for such a method is O ( m / ( T " 2 )
+  2 / " 2 ) , which is worse than the lower bound mentioned above.
Now, the key question is whether we can reduce this deviation by other means.
It turns out that stochastic gradient descent (SGD), where in each step we sample a component function to query randomly, when run with an appropriate learning rate and using averaging, matches the optimal deviation up to constant factors.
We provide a detailed proof of the result in Appendix G.
This provides another justiﬁcation for using SGD instead of gradient descent in practice.
Theorem 4.
For G = O (1) and D = O (1) , let f i be an G -Lipschitz convex cost function for each
i 2 [ m ] , and assume that the optimum of f lies in a ball of radius D .
Let " ,  > 0 be given parameters such that   " / (2 D ) , and T = ⌦ (1 / " 2 ) be a given number of iterations.
Deﬁne the
SGD updates as follows: initialize x 0 = 0 , and for t = 0 , 1 , . .
, T  1 , set x t +
= x t  ⌘ t g i t ( x t ) where i t ⇠ [ n ] uniformly at random.
Under the inexact component gradient oracle (Deﬁnition 4), the average iterate ¯ x T of SGD with stepsize ⌘ = ⇥ ( 1 / ( " T ) ) satisﬁes E f (¯ x T )  inf x 2 dom f f ( x )  " and
E k ¯ x T  ¯ x 0 T k 2 =
O ( 1 / ( T " 2 )
+  2 / " 2 ) , where ¯ x 0 T is the output of an independent run of SGD.
6.2 Optimizing Population Loss (Stochastic Convex Optimization)
The fundamental problem of machine learning is to ﬁnd the solution to the following population (or test) loss minimization problem: minimize F ( x ) =
E ⇠ ⇠ ⌅ f ( x , ⇠ ) , where ⌅ is an unknown distribution on the examples ⇠ , given access to an oracle that sample from ⌅ .
When the function F is convex, this is also known as Stochastic Convex Optimization (SCO).
In this setting, the sampling oracle is a natural source of irreproducibility of optimization methods.
To model this, we consider the stochastic global oracle , inspired by Foster et al.
Deﬁnition 5 (  -bounded stochastic global oracle) .
Given a function F , a  -bounded stochastic global oracle for F is an algorithm that, on each query, draws an independent sample ⇠ ⇠ ⌅ from some distribution ⌅ , and outputs a function f ( · , ⇠ ) : dom F !
R such that for all x 2 dom F , we have
F ( x ) =
E ⇠ ⇠ ⌅ f ( x , ⇠ ) , and, E ⇠ ⇠ ⌅ kr f
( x , ⇠ )  r F ( x ) k 2   2 .
Since in each query, the stochastic global oracle returns the complete speciﬁcation of f ( · , ⇠ ) , an algorithm using such an oracle has access to ( x , f ( x , ⇠ ) , r f ( x , ⇠ ) , r 2 f ( x , ⇠ ) , · · · ) for all x .
Note that the inexact stochastic gradient oracle of Deﬁnition 2 is a weaker form of the stochastic global oracle.
Thus, the key question is: under the powerful stochastic global oracle, can we provide better reproducibility than the lower bounds in Theorem 1?
Surprisingly, the answer to the above question is negative.
That is, despite the stronger oracle setting, there is a function class for which the lower bound on ( " ,  ) -deviation matches that of Theorem 1.
Theorem 5.
Assume that " < 1 / 200 and " .
2 .
Then there exists a family of population costs
F ✓ ( x ) =
E ⇠ ⇠ P ✓ f ✓ ( x , ⇠ ) parametrized by ✓ 2 [1 , 2] , and a  -bounded stochastic global oracle for F ✓ , satisfying the following property.
Suppose that A is any algorithm that for each ✓ 2 [1 , 2] uses at most T queries to the stochastic global oracle and outputs x ✓ T that is " -accurate, i.e.,
E F ✓ ( x ✓ T )  inf x F ✓ ( x )  " .
Then, there exists ✓ bad 2 [1 , 2] such that Var( x ✓ bad T )  ⌦ (  2 / ( T " 2 ) ) .
See Subsection B.2 for a detailed proof.
On the other hand, Theorem 1 also shows that standard SGD with the weaker inexact stochastic gradient oracle achieves this lower bound.
Conclusions
We presented a framework for studying reproducibility in convex optimization, and explored limits of reproducibility for optimizing various function classes under different sources of errors.
For each of these settings, we provide tight lower and upper bounds on reproducibility of ﬁrst order iterative algorithms.
Overall, our results provide the following insights: a) Non-smooth functions can be highly susceptible to even tiny errors which can creep in due to say numerical errors in GPU.
Thus introducing smoothness in deep learning models might help with reproducibility.
b) Generally, gradient descent type methods with small learning rate are more reproducible.
c)
In ﬁnite-sum settings, despite more randomness, SGD is more reproducible than full-batch gradient descent.
The study in this paper is a ﬁrst step towards addressing the challenging problem of reproducibility in a rigorous framework.
So, many important directions remain unexplored.
For example, study of reproducibility of adaptive methods like AdaGrad that are not captured by ( FOI ) is interesting.
We study reproducibility in a strict form where we measure deviation in terms of the learned parameters of the model.
However, in practice, one might care for reproducibility in model predictions only.
So, extending our results to such ML-driven scenarios should be relevant in practice.
Finally, while our lower bounds on reproducibility obviously also hold for non-convex optimization, extension of the upper bounds to non-convex settings and potentially, designing novel rigorous methods in this setting is a fascinating direction for future work.
Acknowledgments and Disclosure of Funding
Kwangjun Ahn was supported by graduate assistantship from the NSF Grant (CAREER: 1846088), the ONR grant (N00014-20-1-2394) and MIT-IBM Watson as well as a Vannevar Bush fellowship from Ofﬁce of the Secretary of Defense.
Part of this work was done while Kwangjun Ahn was visiting the Simons Institute for the Theory of Computing, Berkeley, California.