Original: D EEP OBS: A D EEP L EARNING O PTIMIZER B ENCHMARK S UITE
Revised: D EEP OBS: A D EEP L EARNING O PTIMIZER B ENCHMARK S UITE

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: There is signiﬁcant past and ongoing research on optimization methods for deep learning.
Revised: There is signiﬁcant past and ongoing research on optimization methods for deep learning.

Original: Yet, perhaps surprisingly, there is no generally agreed-upon protocol for the quantitative and reproducible evaluation of such optimizers.
Revised: Yet, perhaps surprisingly, there is no generally agreed-upon protocol for the quantitative and reproducible evaluation of such optimizers.

Original: We suggest routines and benchmarks for stochastic optimization, with special focus on the unique aspects of deep learning, such as stochasticity, tunability and generalization.
Revised: We suggest routines and benchmarks for stochastic optimization, with special focus on the unique aspects of deep learning, such as stochasticity, tunability and generalization.

Original: As the primary contribution, we present D EEP OBS, a Python package of deep learning optimization benchmarks.
Revised: As the primary contribution, we present D EEP OBS, a Python package of deep learning optimization benchmarks.

Original: The package addresses key challenges in the quantitative assessment of stochastic optimizers, and automates most steps of benchmarking.
Revised: The package addresses key challenges in the quantitative assessment of stochastic optimizers, and automates most steps of benchmarking.

Original: The library includes a wide and extensible set of ready-to-use realistic optimization problems, such as training Residual Networks for image classiﬁcation on I MAGE N ET or character-level language prediction models, as well as popular classics like MNIST and CIFAR-10.
Revised: The library includes a wide and extensible set of ready-to-use realistic optimization problems, such as training Residual Networks for image classiﬁcation on I MAGE N ET or character-level language prediction models, as well as popular classics like MNIST and CIFAR-10.

Original: The package also provides realistic baseline results for the most popular optimizers on these test problems, ensuring a fair comparison to the competition when benchmarking new optimizers, and without having to run costly experiments.
Revised: The package also provides realistic baseline results for the most popular optimizers on these test problems, ensuring a fair comparison to the competition when benchmarking new optimizers, and without having to run costly experiments.

Original: It comes with output back-ends that directly produce L A TEX code for inclusion in academic publications.
Revised: It comes with output back-ends that directly produce L A TEX code for inclusion in academic publications.

Original: It is written in
Revised: It supports

Original: T ENSOR F LOW and  available open source.
Revised: T ENSOR F LOW and is available open source.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: As deep learning has become mainstream, research on aspects like architectures (Graves et al., 2014;
Revised: As deep learning has become mainstream, research on aspects like architectures (Graves et al., 2014;

Original: He et al., 2016; Szegedy et al., 2017; Vaswani et al., 2017; Sabour et al., 2017) and hardware (Ovtcharov et al., 2015; Chen et al., 2016; Reagen et al., 2016; Jouppi, 2016) has exploded, and helped professionalize the ﬁeld.
Revised: He et al., 2016; Szegedy et al., 2017; Vaswani et al., 2017; Sabour et al., 2017) and hardware (Ovtcharov et al., 2015; Chen et al., 2016; Reagen et al., 2016; Jouppi, 2016) has exploded, and helped professionalize the ﬁeld.

Original: In comparison, the optimization routines used to train deep nets have arguable changed only little.
Revised: In comparison, the optimization routines used to train deep nets have arguable changed only little.

Original: Comparably simple ﬁrst-order methods like SGD (Robbins &
Revised: Comparably simple ﬁrst-order methods like SGD (Robbins &

Original: Monro, 1951), its momentum variants (M OMENTUM ) (Polyak, 1964; Nesterov, 1983) and A DAM (Kingma & Ba, 2015) remain standards (Goodfellow et al., 2016; Karpathy, 2017).
Revised: Monro, 1951), its momentum variants (M OMENTUM ) (Polyak, 1964; Nesterov, 1983) and A DAM (Kingma & Ba, 2015) remain standards (Goodfellow et al., 2016; Karpathy, 2017).

Original: The low practical relevance of more advanced optimization methods is not for lack of research, though.
Revised: The low practical relevance of more advanced optimization methods is not for lack of research, though.

Original: There is a host of papers proposing new ideas for acceleration of ﬁrst-order methods (Duchi et al., 2011; Tieleman & Hinton, 2012; Zeiler, 2012; Dozat, 2016; Bello et al., 2017; Loshchilov & Hutter, 2017; Reddi et al., 2018), incorporation of second-order information (Martens, 2010; Martens & Grosse, 2015;
Revised: There is a host of papers proposing new ideas for acceleration of ﬁrst-order methods (Duchi et al., 2011; Tieleman & Hinton, 2012; Zeiler, 2012; Dozat, 2016; Bello et al., 2017; Loshchilov & Hutter, 2017; Reddi et al., 2018), incorporation of second-order information (Martens, 2010; Martens & Grosse, 2015;

Original: Botev et al., 2017; Zhang et al., 2017; Chen et al., 2018), and automating optimization (Schaul et al., 2013b; Mahsereci & Hennig, 2017; Rolinek & Martius, 2018), to name just a few.
Revised: Botev et al., 2017; Zhang et al., 2017; Chen et al., 2018), and automating optimization (Schaul et al., 2013b; Mahsereci & Hennig, 2017; Rolinek & Martius, 2018), to name just a few.

Original: One problem is that these methods are algorithmically involved and difﬁcult to reproduce by practitioners.
Revised: One problem is that these methods are algorithmically involved and difﬁcult to reproduce by practitioners.

Original: If they are not provided in packages for popular frameworks like T ENSOR F LOW , P Y T ORCH etc., they get little traction.
Revised: If they are not provided in packages for popular frameworks like T ENSOR F LOW , P Y T ORCH etc., they get little traction.

Original: Another problem, which we hope to address here, is that new optimization routines are often not convincingly compared to simpler alternatives in research papers, so practitioners are left wondering which of the many new choices is the best (and which ones even really work in the ﬁrst place).
Revised: Another problem, which we hope to address here, is that new optimization routines are often not convincingly compared to simpler alternatives in research papers, so practitioners are left wondering which of the many new choices is the best (and which ones even really work in the ﬁrst place).

Original: Designing an empirical protocol for deep learning optimizers is not straightforward, and the corresponding experiments can be time-consuming.
Revised: Designing an empirical protocol for deep learning optimizers is not straightforward, and the corresponding experiments can be time-consuming.

Original: This is partly due to the idiosyncrasies of the domain:
Revised: This is partly due to the idiosyncrasies of the domain:

Original: • Generalization: While the optimization algorithm (should) only ever see the training-set, the practitioner cares about performance of the trained model on the test set.
Revised: • Generalization: While the optimization algorithm (should) only ever see the training-set, the practitioner cares about performance of the trained model on the test set.

Original: Worse, in some important application domains, the optimizer’s loss function is not the objective we ultimately care about.
Revised: Worse, in some important application domains, the optimizer’s loss function is not the objective we ultimately care about.

Original: For instance in image classiﬁcation, the real interest may be in the percentage of correctly labeled images, the accuracy.
Revised: For instance in image classiﬁcation, the real interest may be in the percentage of correctly labeled images, the accuracy.

Original: Since this 0 - 1 loss is infeasible in practice (Marcotte & Savard, 1992), a surrogate loss function is used instead.
Revised: Since this 0 - 1 loss is infeasible in practice (Marcotte & Savard, 1992), a surrogate loss function is used instead.

Original: So which score should actually be presented in a comparison of optimizers?
Revised: So which score should actually be presented in a comparison of optimizers?

Original: Train loss, because that is what the optimizer actually works on; test loss, because an over-ﬁtting optimizer is useless, or test accuracy, because that’s what the human user cares about?
Revised: Train loss, because that is what the optimizer actually works on; test loss, because an over-ﬁtting optimizer is useless, or test accuracy, because that’s what the human user cares about?

Original: • Stochasticity:
Revised: • Stochasticity:

Original: Sub-sampling (batching) the data-set to compute estimates of the loss function and its gradient introduces stochasticity.
Revised: Sub-sampling (batching) the data-set to compute estimates of the loss function and its gradient introduces stochasticity.

Original: Thus, when an optimizer is run only once on a given problem, its performance may be misleading due to random ﬂuctuations.
Revised: Thus, when an optimizer is run only once on a given problem, its performance may be misleading due to random ﬂuctuations.

Original: The same stochasticity also causes many optimization algorithms to have one or several tuning parameters (learning rates, etc.).
Revised: The same stochasticity also causes many optimization algorithms to have one or several tuning parameters (learning rates, etc.).

Original: How should an optimizer with two free parameter be compared in a fair way with one that has only one, or even no free parameters?
Revised: How should an optimizer with two free parameter be compared in a fair way with one that has only one, or even no free parameters?

Original: • Realistic Settings, Fair Competition: There is a widely-held belief that popular standards like MNIST and CIFAR-10 are too simplistic to serve as a realistic place-holder for a contemporary combination of large-scale data set and architecture.
Revised: • Realistic Settings, Fair Competition: There is a widely-held belief that popular standards like MNIST and CIFAR-10 are too simplistic to serve as a realistic place-holder for a contemporary combination of large-scale data set and architecture.

Original: While this worry is not unfounded, researchers, ourselves included, have sometimes found it hard to satisfy the demands of reviewers for ever new data sets and architectures.
Revised: While this worry is not unfounded, researchers, ourselves included, have sometimes found it hard to satisfy the demands of reviewers for ever new data sets and architectures.

Original: Finding and preparing such datasets and building a reasonable architecture for them is time-consuming for researchers who want to focus on their novel algorithm.
Revised: Finding and preparing such data sets and building a reasonable architecture for them is time-consuming for researchers who want to focus on their novel algorithm.

Original: Even when this is done, one then has to not just run one’s own algorithm, but also various competing baselines, like SGD, M OMENTUM , A DAM , etc.
Revised: Even when this is done, one then has to not just run one’s own algorithm, but also various competing baselines, like SGD, M OMENTUM , A DAM , etc.

Original: This step does not just cost time, it also poses a risk of bias, as the competition invariably receives less care than one’s own method.
Revised: This step does not just cost time, it also poses a risk of bias, as the competition invariably receives less care than one’s own method.

Original: Reviewers and readers can never be quite sure that an author has not tried a bit too much to make their own method look good, either by choosing a convenient training problem, or by neglecting to tune the competition.
Revised: Reviewers and readers can never be quite sure that an author has not tried a bit too much to make their own method look good, either by choosing a convenient training problem, or by neglecting to tune the competition.

Original: To address these problems we propose an extensible, open-source benchmark speciﬁcally for optimization methods on deep learning architectures.
Revised: To address these problems we propose an extensible, open-source benchmark speciﬁcally for optimization methods on deep learning architectures.

Original: We make the following three contributions:
Revised: We make the following three contributions:

Original: • A protocol for benchmarking stochastic optimizers.
Revised: • A protocol for benchmarking stochastic optimizers.

Original: Section 2 discusses and recommends best practices for the evaluation of deep learning optimizers.
Revised: Section 2 discusses and recommends best practices for the evaluation of deep learning optimizers.

Original: We deﬁne three key performance indicators: ﬁnal performance, speed, and tunability, and suggest means of measuring all three in practice.
Revised: We deﬁne three key performance indicators: ﬁnal performance, speed, and tunability, and suggest means of measuring all three in practice.

Original: We provide evidence that it is necessary to show the results of multiple runs in order to get a realistic assessment.
Revised: We provide evidence that it is necessary to show the results of multiple runs in order to get a realistic assessment.

Original: Finally, we strongly recommend reporting both loss and accuracy, for both training and test set, when demonstrating a new optimizer as there is no obvious way those four learning curves are connected in general.
Revised: Finally, we strongly recommend reporting both loss and accuracy, for both training and test set, when demonstrating a new optimizer as there is no obvious way those four learning curves are connected in general.

Original: • D EEP OBS 1 , a deep learning optimizer benchmark suite.
Revised: • D EEP OBS 1 , a deep learning optimizer benchmark suite.

Original: We have distilled the above ideas into an open-source python package, written in T ENSOR F LOW (Abadi et al., 2015), which automates most of the steps presented in section 2.
Revised: We have distilled the above ideas into an open-source python package, written in T ENSOR F LOW (Abadi et al., 2015), which automates most of the steps presented in section 2.

Original: The package currently provides over twenty off-the-shelf test problems across four application domains, including image classiﬁcation and natural language processing, and this collection can be extended and adapted as the ﬁeld makes progress.
Revised: The package currently provides over twenty off-the-shelf test problems across four application domains, including image classiﬁcation and natural language processing, and this collection can be extended and adapted as the ﬁeld makes progress.

Original: The test problems range in complexity from stochastic two dimensional functions to contemporary deep neural networks capable of delivering near state-of-the-art results on datasets such as I MAGE N ET .
Revised: The test problems range in complexity from stochastic two dimensional functions to contemporary deep neural networks capable of delivering near state-of-the-art results on data sets such as I MAGE N ET .

Original: The package is easy to install in python, using the pip toolchain.
Revised: The package is easy to install in python, using the pip toolchain.

Original: It automatically downloads datasets, sets up models, and provides a back-end to automatically produce L A TEX code that can directly be included in academic publications.
Revised: It automatically downloads data sets, sets up models, and provides a back-end to automatically produce L A TEX code that can directly be included in academic publications.

Original: This automation does not just save time, it also helps researchers to create reproducible, comparable, and interpretable results.
Revised: This automation does not just save time, it also helps researchers to create reproducible, comparable, and interpretable results.

Original: • Benchmark of popular optimizers From the collection of test problems, two sets, of four simple (“small”) and four more demanding (“large”) problems, respectively, are selected as a core set of benchmarks.
Revised: • Benchmark of popular optimizers From the collection of test problems, two sets, of four simple (“small”) and four more demanding (“large”) problems, respectively, are selected as a core set of benchmarks.

Original: Researchers can design their algorithm in rapid iterations on the simpler set, then test on the more demanding set.
Revised: Researchers can design their algorithm in rapid iterations on the simpler set, then test on the more demanding set.

Original: We argue that this protocol saves time, while also reducing the risk of over-ﬁtting in the algorithm design loop.
Revised: We argue that this protocol saves time, while also reducing the risk of over-ﬁtting in the algorithm design loop.

Original: The package also provides realistic baselines results for the most popular optimizers on those test problems.
Revised: The package also provides realistic baselines results for the most popular optimizers on those test problems.

Original: In Section 4 we report on the performance of SGD, SGD with momentum (M OMENTUM ) and A DAM on the small and large benchmarks (this also demonstrates the output of the benchmark).
Revised: In Section 4 we report on the performance of SGD, SGD with momentum (M OMENTUM ) and A DAM on the small and large benchmarks (this also demonstrates the output of the benchmark).

Original: For each optimizer we perform an exhaustive but realistic hyperparameter search.
Revised: For each optimizer we perform an exhaustive but realistic hyperparameter search.

Original: The best performing results are provided with D EEP OBS and can be used as a fair performance metric for new optimizers without the need to compute these baselines again.
Revised: The best performing results are provided with D EEP OBS and can be used as a fair performance metric for new optimizers without the need to compute these baselines again.

Original: We invite the authors of other algorithms to add their own method to the benchmark (via a git pull-request).
Revised: We invite the authors of other algorithms to add their own method to the benchmark (via a git pull-request).

Original: We hope that the benchmark will offer a common platform, allowing researchers to publicise their algorithms, giving practitioners a clear view on the state of the art, and helping the ﬁeld to more rapidly make progress.
Revised: We hope that the benchmark will offer a common platform, allowing researchers to publicise their algorithms, giving practitioners a clear view on the state of the art, and helping the ﬁeld to more rapidly make progress.

Original: 1. R ELATED WORKS
Revised: 1. R ELATED WORKS

Original: To our knowledge, there is currently no commonly accepted benchmark for optimization algorithms that is well adapted to the deep learning setting.
Revised: To our knowledge, there is currently no commonly accepted benchmark for optimization algorithms that is well adapted to the deep learning setting.

Original: This impression is corroborated by a more or less random sample of recent research papers on deep learning optimization (Duchi et al., 2011; Zeiler, 2012; Kingma & Ba, 2015; Martens & Grosse, 2015; Dozat, 2016; Bello et al., 2017; Loshchilov & Hutter, 2017; Reddi et al., 2018), whose empirical sections follow no joint standard (beyond a popularity of the MNIST dataset).
Revised: This impression is corroborated by a more or less random sample of recent research papers on deep learning optimization (Duchi et al., 2011; Zeiler, 2012; Kingma & Ba, 2015; Martens & Grosse, 2015; Dozat, 2016; Bello et al., 2017; Loshchilov & Hutter, 2017; Reddi et al., 2018), whose empirical sections follow no joint standard (beyond a popularity of the MNIST data set).

Original: There are a number of existing benchmarks for deep learning as such.
Revised: There are a number of existing benchmarks for deep learning as such.

Original: However, they do not focus on the optimizer.
Revised: However, they do not focus on the optimizer.

Original: Instead, they are either framework or hardwarespeciﬁc, or cover deep learning as a holistic process, wrapping together architecture, hardware and training procedure, The following are among most popular ones:
Revised: Instead, they are either framework or hardwarespeciﬁc, or cover deep learning as a holistic process, wrapping together architecture, hardware and training procedure, The following are among most popular ones:

Original: DAWNBench The task in this challenge is to train a model for I MAGE N ET , CIFAR-10 or SQ U AD (Rajpurkar et al., 2018) as quickly as possible to a speciﬁed validation accuracy, tuning the entire tool-chain from architecture to hardware and optimizer (Coleman et al., 2017).
Revised: DAWNBench The task in this challenge is to train a model for I MAGE N ET , CIFAR-10 or SQ U AD (Rajpurkar et al., 2018) as quickly as possible to a speciﬁed validation accuracy, tuning the entire tool-chain from architecture to hardware and optimizer (Coleman et al., 2017).

Original: MLPerf is another holistic benchmark similar to DAWNBench.
Revised: MLPerf is another holistic benchmark similar to DAWNBench.

Original: It has two different rule sets; only the ‘open’ set allows a choice of optimization algorithm (MLPerf, 2018).
Revised: It has two different rule sets; only the ‘open’ set allows a choice of optimization algorithm (MLPerf, 2018).

Original: Deep Learning Frameworks (Comparison) compares runtimes of different high-level frameworks (Microsoft Machine Learning, 2018).
Revised: Deep Learning Frameworks (Comparison) compares runtimes of different high-level frameworks (Microsoft Machine Learning, 2018).

Original: DLBS is a benchmark focused on the performance of deep learning models on various hardware systems with various software (Hewlett Packard Enterprise, 2017).
Revised: DLBS is a benchmark focused on the performance of deep learning models on various hardware systems with various software (Hewlett Packard Enterprise, 2017).

Original: DeepBench tests the speed of hardware for the low-level operations of deep learning, like matrix products and convolutions (Baidu Research, 2016).
Revised: DeepBench tests the speed of hardware for the low-level operations of deep learning, like matrix products and convolutions (Baidu Research, 2016).

Original: Fathom is another hardware-centric benchmark, which among other things assesses how computational resources are spent (Adolf et al., 2016).
Revised: Fathom is another hardware-centric benchmark, which among other things assesses how computational resources are spent (Adolf et al., 2016).

Original: TBD focuses on the performance of three deep learning frameworks (Zhu et al., 2018).
Revised: TBD focuses on the performance of three deep learning frameworks (Zhu et al., 2018).

Original: None of these benchmarks are good test beds for optimization research.
Revised: None of these benchmarks are good test beds for optimization research.

Original: Schaul et al.
Revised: Schaul et al.

Original: (2013a) deﬁned unit tests for stochastic optimization.
Revised: (2013a) deﬁned unit tests for stochastic optimization.

Original: In contrast to the present work, they focus on small-scale problems like quadratic bowls and cliffs.
Revised: In contrast to the present work, they focus on small-scale problems like quadratic bowls and cliffs.

Original: In the context of deep learning, these problems provide unit tests, but do not give a realistic impression of an algorithm’s performance in practice.
Revised: In the context of deep learning, these problems provide unit tests, but do not give a realistic impression of an algorithm’s performance in practice.

Original:  B ENCHMARKING DEEP LEARNING OPTIMIZERS
Revised:  B ENCHMARKING DEEP LEARNING OPTIMIZERS

Original: This section expands the discussion from section 1 of design desiderata for a good benchmark protocol, and proposes ways to nevertheless arrive at an informative, fair, and reproducible benchmark.
Revised: This section expands the discussion from section 1 of design desiderata for a good benchmark protocol, and proposes ways to nevertheless arrive at an informative, fair, and reproducible benchmark.

Original: 2. S TOCHASTICITY
Revised: 2. S TOCHASTICITY

Original: The optimizer’s performance in a concrete training run is noisy, due to the random sampling of mini-batches and initial parameters.
Revised: The optimizer’s performance in a concrete training run is noisy, due to the random sampling of mini-batches and initial parameters.

Original: There is an easy remedy, which nevertheless is not universally adhered to: Optimizers should be run on the same problem repeatedly with different random seeds, and all relevant quantities should be reported as mean and standard deviation of these samples.
Revised: There is an easy remedy, which nevertheless is not universally adhered to: Optimizers should be run on the same problem repeatedly with different random seeds, and all relevant quantities should be reported as mean and standard deviation of these samples.

Original: This allows judging the statistical signiﬁcance of small performance differences between optimizers, and exposes the “variability” of performance of an optimizer on any given problem.
Revised: This allows judging the statistical signiﬁcance of small performance differences between optimizers, and exposes the “variability” of performance of an optimizer on any given problem.

Original: The obvious reason why researchers are reluctant to follow this standard is that it requires substantial computational effort.
Revised: The obvious reason why researchers are reluctant to follow this standard is that it requires substantial computational effort.

Original: D EEP OBS alleviates this issue in two ways: It provides functionality to conveniently run multiple runs of the same setting with different seeds.
Revised: D EEP OBS alleviates this issue in two ways: It provides functionality to conveniently run multiple runs of the same setting with different seeds.

Original: More importantly, it provides stored baselines of popular optimizers, freeing computational resources to collect statistics rather than baselines.
Revised: More importantly, it provides stored baselines of popular optimizers, freeing computational resources to collect statistics rather than baselines.

Original: 2. C HOICE OF PERFORMANCE METRIC
Revised: 2. C HOICE OF PERFORMANCE METRIC

Original: Training a machine learning system is more than a pure optimization problem.
Revised: Training a machine learning system is more than a pure optimization problem.

Original: The optimizers’ immediate objective is training loss , but the users’ interest is in generalization performance, as estimated on a held-out test set.
Revised: The optimizers’ immediate objective is training loss , but the users’ interest is in generalization performance, as estimated on a held-out test set.

Original: It has been observed repeatedly that in deep learning, different optimizers of similar training-set performance can have surprisingly different generalization (e.g.
Revised: It has been observed repeatedly that in deep learning, different optimizers of similar training-set performance can have surprisingly different generalization (e.g.

Original: Wilson et al.
Revised: Wilson et al.

Original: Moreover, the loss function is regularly just a surrogate for the metric the user is ultimately interested in.
Revised: Moreover, the loss function is regularly just a surrogate for the metric the user is ultimately interested in.

Original: In classiﬁcation problems, for example, we are interested in classiﬁcation accuracy, but this is infeasible to optimize directly.
Revised: In classiﬁcation problems, for example, we are interested in classiﬁcation accuracy, but this is infeasible to optimize directly.

Original: Thus, there are up to four relevant metrics to consider: training loss, test loss, training accuracy and test accuracy.
Revised: Thus, there are up to four relevant metrics to consider: training loss, test loss, training accuracy and test accuracy.

Original: We strongly recommend reporting all four of these to give a comprehensive assessment of a deep learning optimizer.
Revised: We strongly recommend reporting all four of these to give a comprehensive assessment of a deep learning optimizer.

Original: For hyperparameter tuning, we use test accuracy or, if that is not available, test loss, as the criteria.
Revised: For hyperparameter tuning, we use test accuracy or, if that is not available, test loss, as the criteria.

Original: We also use them as the performance metrics in Table 2.
Revised: We also use them as the performance metrics in Table 2.

Original: For empirical plots, many authors compute train loss (or accuracy) only on mini-batches of data, since these are computed during training anyway.
Revised: For empirical plots, many authors compute train loss (or accuracy) only on mini-batches of data, since these are computed during training anyway.

Original: But these mini-batch quantities are subject to signiﬁcant noise.
Revised: But these mini-batch quantities are subject to signiﬁcant noise.

Original: To get a decent estimate of the training-set performance, whenever we evaluate on the test set, we also evaluate on a larger chunk of training data, which we call a train eval set.
Revised: To get a decent estimate of the training-set performance, whenever we evaluate on the test set, we also evaluate on a larger chunk of training data, which we call a train eval set.

Original: In addition to providing a more accurate estimate, this allows us to “switch” the architecture to evaluation mode (e.g. dropout is not used during evaluation).
Revised: In addition to providing a more accurate estimate, this allows us to “switch” the architecture to evaluation mode (e.g. dropout is not used during evaluation).

Original: 2. M EASURING SPEED
Revised: 2. M EASURING SPEED

Original: Relevant in practice is not only the quality of a solution, but also the time required to reach it.
Revised: Relevant in practice is not only the quality of a solution, but also the time required to reach it.

Original: A fast optimizer that ﬁnds a decent albeit imperfect solution using a fraction of other methods’ resources can be very relevant in practice.
Revised: A fast optimizer that ﬁnds a decent albeit imperfect solution using a fraction of other methods’ resources can be very relevant in practice.

Original: Unfortunately, since learning curves have no parametric form, there is no uniquely correct way to deﬁne “time to convergence”.
Revised: Unfortunately, since learning curves have no parametric form, there is no uniquely correct way to deﬁne “time to convergence”.

Original: In D EEP OBS, we take a pragmatic approach and measure the time it takes to reach an “acceptable” convergence performance , which is individually deﬁned for each test problem from the baselines SGD, M OMENTUM and A DAM each with their best hyperparameter setting.
Revised: In D EEP OBS, we take a pragmatic approach and measure the time it takes to reach an “acceptable” convergence performance , which is individually deﬁned for each test problem from the baselines SGD, M OMENTUM and A DAM each with their best hyperparameter setting.

Original: Arguably the most relevant measure of speed would be the wall-clock time to reach this convergence performance.
Revised: Arguably the most relevant measure of speed would be the wall-clock time to reach this convergence performance.

Original: However, wall-clock runtime has well-known drawbacks, such as dependency on hardware or weak reproducibility.
Revised: However, wall-clock runtime has well-known drawbacks, such as dependency on hardware or weak reproducibility.

Original: So many authors report performance against gradient evaluations, since these often dominate the total computational costs.
Revised: So many authors report performance against gradient evaluations, since these often dominate the total computational costs.

Original: However, this can hide large per-iteration overhead.
Revised: However, thiscan hide large per-iteration overhead.

Original: We recommend ﬁrst measuring wall-clock time of both the new competitor and SGD on one of the small test problems for a few iterations, and computing their ratio.
Revised: We recommend ﬁrst measuring wall-clock time of both the new competitor and SGD on one of the small test problems for a few iterations, and computing their ratio.

Original: This computation, which can be done automatically using D EEP OBS, can be done sequentially on the same hardware.
Revised: This computation, which can be done automatically using D EEP OBS, can be done sequentially on the same hardware.

Original: One can then report performance against the products of iterations and per-iteration cost relative to
Revised: One can then report performance against the products of iterations and per-iteration cost relative to

Original: SGD.
Revised: SGD.

Original: For many ﬁrst-order optimization methods, such as SGD, M OMENTUM or A DAM , the choice of hyperparameters does not affect the runtime of the algorithm.
Revised: For many ﬁrst-order optimization methods, such as SGD, M OMENTUM or A DAM , the choice of hyperparameters does not affect the runtime of the algorithm.

Original: However, more evolved optimization methods, e.g. ones that dynamically estimate the Hessian, the hyperparameters can inﬂuence the runtime signiﬁcantly.
Revised: However, more evolved optimization methods, e.g. ones that dynamically estimate the Hessian, the hyperparameters can inﬂuence the runtime signiﬁcantly.

Original: In those cases, it is suggested to repeat the runtime estimate for different hyperparameters.
Revised: In those cases, it is suggested to repeat the runtime estimate for different hyperparameters.

Original: 2. H YPERPARAMETER TUNING
Revised: 2. H YPERPARAMETER TUNING

Original: Almost all deep learning optimizers expose tunable hyperparameters, e.g., step sizes or averaging constants.
Revised: Almost all deep learning optimizers expose tunable hyperparameters, e.g., step sizes or averaging constants.

Original: The ease of tuning these hyperparameters is a relevant characteristic of an optimization method.
Revised: The ease of tuning these hyperparameters is a relevant characteristic of an optimization method.

Original: How does one “fairly” compare optimizers with tunable hyperparameters?
Revised: How does one “fairly” compare optimizers with tunable hyperparameters?

Original: A full analysis of the effects of an optimizer’s hyperparameters on its performance and speed is tedious, especially since they often interact.
Revised: A full analysis of the effects of an optimizer’s hyperparameters on its performance and speed is tedious, especially since they often interact.

Original: Even a simpler sensitivity analysis requires a large number of optimization runs, which are infeasible for most users.
Revised: Even a simpler sensitivity analysis requires a large number of optimization runs, which are infeasible for most users.

Original: Such analyses also do not take into account if hyperparameters have default values that work for almost all optimization problems and therefore require no tuning in general.
Revised: Such analyses also do not take into account if hyperparameters have default values that work for almost all optimization problems and therefore require no tuning in general.

Original: Instead we recommend that authors ﬁnd and report the bestperforming hyperparameters for each test problem.
Revised: Instead we recommend that authors ﬁnd and report the bestperforming hyperparameters for each test problem.

Original: Since D EEP OBS covers multiple test problems, the spread of these best choices gives a good impression of the required tuning.
Revised: Since D EEP OBS covers multiple test problems, the spread of these best choices gives a good impression of the required tuning.

Original: Additionally, we suggest reporting the relative performance of the hyperparameter settings used during this tuning process (Figure 3 shows an example).
Revised: Additionally, we suggest reporting the relative performance of the hyperparameter settings used during this tuning process (Figure 3 shows an example).

Original: Doing so yields a characterization of tunability without additional computations.
Revised: Doing so yields a characterization of tunability without additional computations.

Revised: For the baselines presented in this paper, we chose a simple log-grid search to tune the learning rate.

Revised: While this is certainly not an optimal tuning method, and more sophisticated methods exists (e.g.

Revised: (Bergstra & Bengio, 2012), (Snoek et al., 2012)), it is nevertheless used often in practice and reveals interesting properties about the optimizers and their tunability.

Revised: Other tuning methods can be used with D EEP OBS however, this would require recomputing the baselines as well.

Original: D EEP OBS supports authors in adhering to good scientiﬁc practice by removing various moral hazards.
Revised: D EEP OBS supports authors in adhering to good scientiﬁc practice by removing various moral hazards.

Original: The baseline results for popular optimizers (whose hyperparameters have been tuned by us or, in the future, the very authors of the competing methods) avoid “starving” the competition of attention.
Revised: The baseline results for popular optimizers (whose hyperparameters have been tuned by us or, in the future, the very authors of the competing methods) avoid “starving” the competition of attention.

Revised: When using different hyperparameter tuning methods, it is necessary to allocate the same computational budget for all methods in particular when comparing optimization methods of varying number of hyperparameters.

Original: The ﬁxed set of test problems provided by the benchmark makes it impossible to (knowingly or subconsciously) cherry-pick problems tuned to a new method.
Revised: The ﬁxed set of test problems provided by the benchmark makes it impossible to (knowingly or subconsciously) cherry-pick problems tuned to a new method.

Original: And ﬁnally, the fact that the benchmark spreads over multiple such problem sets constitutes a mild but natural barrier to “overﬁt” the optimizer method to established data sets and architectures (like MNIST).
Revised: And ﬁnally, the fact that the benchmark spreads over multiple such problem sets constitutes a mild but natural barrier to “overﬁt” the optimizer method to established data sets and architectures (like MNIST).

Original:  B ENCHMARK SUITE OVERVIEW
Revised:  B ENCHMARK SUITE OVERVIEW

Original: [Figure]
Revised: [Figure]

Revised: Model Loading

Revised: Runners

Revised: Baseline Results

Revised: RuntimeEstimation

Original: Visualization
Revised: Visualization

Original: Data [Figure]
Revised: Data Loading

Original: Model Loading
Revised: Model Loading

Original: Run Scripts
Revised: Runners

Original: Baseline Results
Revised: Baseline Results

Original: [Figure]
Revised: RuntimeEstimation

Original: Visualization
Revised: Visualization

Original: Pre-processed and batched data.
Revised: Pre-processed and batched data.

Original: Losses and ac- curacy of a deep learning model.
Revised: Losses and ac- curacy of a deep learning model.

Original: Optimization per- formance of an optimizer on a spe- ciﬁc test problem.
Revised: Optimization per- formance of an optimizer on a spe- ciﬁc test problem.

Original: Performances re-
Revised: Performances re-

Original: sults of the most popular optimizers.
Revised: sults of the most popular optimizers.

Original: .tex
Revised: .tex

Original: ﬁles of learn- ing curves for new optimizer and the baselines.
Revised: ﬁles of learn- ing curves for new optimizer and the baselines.

Original: signals data packaged with D EEP OBS and denotes parts provided through template scripts.

Original: D EEP OBS provides the full stack required for rapid, reliable, and reproducible benchmarking of deep learning optimizers.
Revised: D EEP OBS provides the full stack required for rapid, reliable, and reproducible benchmarking of deep learning optimizers.

Original: At the lowest level, a data loading ( § 3.1) module automatically loads and preprocesses data sets  from the net.
Revised: At the lowest level, a data loading ( § 3.1) module automatically loads and preprocesses data sets downloaded from the net.

Original: These are combined with a list of models ( § 3.2) to deﬁne test problems.
Revised: These are combined with a list of models ( § 3.2) to deﬁne test problems.

Original: At the core of the library, run scripts ( § 3.3) take care of the actual training, and log a multitude of statistics, e.g., training loss or test accuracy.
Revised: At the core of the library, runners ( § 3.3) take care of the actual training, and log a multitude of statistics, e.g., training loss or test accuracy.

Original: Baselines ( § 3.4) are provided for a collection of competitors.
Revised: Baselines ( § 3.4) are provided for a collection of competitors.

Original: They currently include the popular choices SGD (raw, and with M OMENTUM ) and A DAM , but we invite authors of other methods to contribute their own.
Revised: They currently include the popular choices SGD (raw, and with M OMENTUM ) and A DAM , but we invite authors of other methods to contribute their own.

Original: The visualization ( § 3.6) script maps the results to L A TEX output.
Revised: The visualization ( § 3.6) script maps the results to L A TEX output.

Original: Future releases of D EEP OBS will include a version number that follows the pattern MAJOR.MINOR.PATCH, where MAJOR versions will differ in the selection of the benchmark sets, MINOR versions signify changes that could affect the results.
Revised: Future releases of D EEP OBS will include a version number that follows the pattern MAJOR.MINOR.PATCH, where MAJOR versions will differ in the selection of the benchmark sets, MINOR versions signify changes that could affect the results.

Original: PATCHES will not affect the benchmark results.
Revised: PATCHES will not affect the benchmark results.

Original: All results obtained with the same MAJOR.MINOR version of D EEP OBS will be directly comparable, all results with the same MAJOR version will compare results on the same problems.
Revised: All results obtained with the same MAJOR.MINOR version of D EEP OBS will be directly comparable, all results with the same MAJOR version will compare results on the same problems.

Original: We now give a brief overview of the functionality; the full documentation can be found online.
Revised: We now give a brief overview of the functionality; the full documentation can be found online.

Original: 3. D ATA LOADING
Revised: 3. D ATA LOADING

Original: D EEP OBS can automatically download and pre-process all necessary data sets.
Revised: D EEP OBS can automatically download and pre-process all necessary data sets.

Original: 3 Excluding I MAGE N ET , the downloaded data sets require less than one GB of disk space.
Revised: 3 Excluding I MAGE N ET , the downloaded data sets require less than one GB of disk space.

Original: The D EEP OBS data loading module then performs all necessary processing of the data sets to return inputs and outputs for the deep learning model (e.g. images and labels for image classiﬁcation).
Revised: The D EEP OBS data loading module then performs all necessary processing of the data sets to return inputs and outputs for the deep learning model (e.g. images and labels for image classiﬁcation).

Original: This processing includes splitting, shufﬂing, batching and data augmentation.
Revised: This processing includes splitting, shufﬂing, batching and data augmentation.

Original: The data loading module can also be used to build new deep learning models that are not (yet) part of D EEP OBS.
Revised: The data loading module can also be used to build new deep learning models that are not (yet) part of D EEP OBS.

Original: 3. M ODELS
Revised: 3. M ODELS

Original: Together, data set and model deﬁne a loss function and thus an optimization problem.
Revised: Together, data set and model deﬁne a loss function and thus an optimization problem.

Original: Tableprovides an overview of the data sets and models included in D EEP OBS.
Revised: Tableprovides an overview of the data sets and models included in D EEP OBS.

Original: We selected problems for diversity of task as well as the difﬁculty of the optimization problem itself.
Revised: We selected problems for diversity of task as well as the difﬁculty of the optimization problem itself.

Original: The list includes popular image classiﬁcation models on data sets like MNIST, CIFAR-10 or I MAGE N ET , but also models for natural language processing and generative models.
Revised: The list includes popular image classiﬁcation models on data sets like MNIST, CIFAR-10 or I MAGE N ET , but also models for natural language processing and generative models.

Original: Additionally, three two-dimensional problems and an n -dimensional quadratic problem are included.
Revised: Additionally, three two-dimensional problems and an ill-conditioned quadratic problem are included.

Original: These simple tests can be used as illustrative toy problems to highlight properties of an algorithm and perform sanity-checks.
Revised: These simple tests can be used as illustrative toy problems to highlight properties of an algorithm and perform sanity-checks.

Original: Over time, we plan to expand this list when hardware and research progress renders small problems out of date, and introduces new research directions and more challenging problems. 
Revised: Over time, we plan to expand this list when hardware and research progress renders small problems out of date, and introduces new research directions and more challenging problems. , natural language processing

Revised: or problems where the loss function is given explicitly .

Revised: Test problems marked in and are part of the small and large benchmark set, respectively.

Original: [Table] [Table]
Revised: [Figure] [Table]

Original: 3. R UN SCRIPTS
Revised: 3. R UNNERS

Original: The run scripts of the D EEP OBS package handle training and the logging of statistics measuring the optimizers performance.
Revised: The runners of the D EEP OBS package handle training and the logging of statistics measuring the optimizers performance.

Original:  We provide blueprints for the most popular optimizers that can be adapted to new optimization algorithms (often a one-line change sufﬁces). Authors who want their optimizers included in the benchmark should provide the corresponding run scripts via git pull-request.
Revised: For optimizers following the standard TensorFlow optimizer API it is enough to provide the runners with a list of the optimizer’s hyperparameters. We provide a template for this, as well as an example of including a more sophisticated optimizer that can’t be described as a subclass of the TensorFlow optimizer API.

Original: 3. B ASELINES
Revised: 3. B ASELINES

Original: D EEP OBS also provides realistic baselines results for, currently, the three most popular optimizers in deep learning, SGD, M OMENTUM , and A DAM .
Revised: D EEP OBS also provides realistic baselines results for, currently, the three most popular optimizers in deep learning, SGD, M OMENTUM , and A DAM .

Original: These allow comparing a newly developed algorithm to the competition without computational overhead, and without risk of conscious or unconscious bias against the competition.
Revised: These allow comparing a newly developed algorithm to the competition without computational overhead, and without risk of conscious or unconscious bias against the competition.

Original: Section 4 describes how these baselines were constructed and discusses their performance.
Revised: Section 4 describes how these baselines were constructed and discusses their performance.

Original: Baselines for further optimizers will be added when authors provide their run-scripts, assuming the methods perform competitively.
Revised: Baselines for further optimizers will be added when authors provide the optimizer’s code, assuming the method perform competitively.

Original: Currently, baselines are available for all test problems in the small and large benchmark set; we plan to provide baselines for the full set of models in the near future.
Revised: Currently, baselines are available for all test problems in the small and large benchmark set; we plan to provide baselines for the full set of models in the near future.

Original: 3. E STIMATE RUNTIME
Revised: 3. E STIMATE RUNTIME

Original: D EEP OBS provides an option to quickly estimate the runtime overhead of a new optimization method compared to SGD.
Revised: D EEP OBS provides an option to quickly estimate the runtime overhead of a new optimization method compared to SGD.

Original: It measures the ratio of wall-clock time between the new optimizer and SGD.
Revised: It measures the ratio of wall-clock time between the new optimizer and SGD.

Original: By default this ratio is measured on ﬁve runs each, for three epochs, on a fully connected network on
Revised: By default this ratio is measured on ﬁve runs each, for three epochs, on a fully connected network on

Original: MNIST.
Revised: MNIST.

Original: However, this can be adapted to a setting which fairly evaluates the new optimizer, as some optimizers might have a high initial cost that amortizes over many epochs.
Revised: However, this can be adapted to a setting which fairly evaluates the new optimizer, as some optimizers might have a high initial cost that amortizes over many epochs.

Original: 3. V ISUALIZATIONS
Revised: 3. V ISUALIZATIONS

Original: The D EEP OBS visualization module reduces the overhead for the preparation of results, and simultaneously standardizes the presentation, making it possible to include a comparably large amount of information in limited space.
Revised: The D EEP OBS visualization module reduces the overhead for the preparation of results, and simultaneously standardizes the presentation, making it possible to include a comparably large amount of information in limited space.

Original: The module produces .tex
Revised: The module produces .tex

Original: ﬁles with pgfplots -code for all learning curves for the proposed optimizer as well as the most relevant baselines (section 4 includes an example of this output).
Revised: ﬁles with pgfplots -code for all learning curves for the proposed optimizer as well as the most relevant baselines (section 4 includes an example of this output).

Original:  I NSIGHTS FROM THE BASELINES
Revised:  I NSIGHTS FROM THE BASELINES

Original: For the baseline results provided with D EEP OBS, we evaluate three popular deep learning optimizers (SGD, M OMENTUM and A DAM ) on the eight test problems that are part of the small (problems P1 to
Revised: For the baseline results provided with D EEP OBS, we evaluate three popular deep learning optimizers (SGD, M OMENTUM and A DAM ) on the eight test problems that are part of the small (problems P1 to

Original: P4) and large (problems P5 to P8) benchmark set (cf.
Revised: P4) and large (problems P5 to P8) benchmark set (cf.

Original: Table 1 or the appendix).
Revised: Table 1 or the appendix).

Original: The experiments were done with version 1.0 of D EEP OBS.
Revised: The experiments were done with version 1.1.0 of D EEP OBS.

Original: All experiments used 0 .
Revised: All experiments used 0 .

Original: 99 for the M OMENTUM parameter and default parameters for A DAM ( β 1 = 0 .
Revised: 99 for the M OMENTUM parameter and default parameters for A DAM ( β 1 = 0 .

Original: 9 , β 2 = 0 .
Revised: 9 , β 2 = 0 .

Original: 999 , (cid:15) =
Revised: 999 , (cid:15) =

Original: 10 − 8 ).
Revised: 10 − 8 ).

Original: The learning rate α was tuned for each optimizer and test problem individually, by evaluating on a logarithmic grid from α min = 10 −to α max = 10 2 with 36 samples.
Revised: The learning rate α was tuned for each optimizer and test problem individually, by evaluating on a logarithmic grid from α min = 10 −to α max = 10 2 with 36 samples.

Original: Once the best learning rate has been determined, we run those settings ten times with different random seeds.
Revised: Once the best learning rate has been determined, we run those settings ten times with different random seeds.

Original: While we are using a log grid search, researchers are free to use any other hyperparameter tuning method, however this would require re-running the baselines as well.
Revised: While we are using a log grid search, researchers are free to use any other hyperparameter tuning method, however this would require re-running the baselines as well.

Original: Figure 2 shows the learning curves of the eight problems in the small and large benchmark set.
Revised: Figure 2 shows the learning curves of the eight problems in the small and large benchmark set.

Original: Table 2 summarizes the results from both benchmark sets.
Revised: Table 2 summarizes the results from both benchmark sets.

Original: We focus on three main observations, which corroborate widely-held beliefs and support the case for an extensive and standardized benchmark.
Revised: We focus on three main observations, which corroborate widely-held beliefs and support the case for an extensive and standardized benchmark.

Original: There is no optimal optimizer for all test problems.
Revised: There is no optimal optimizer for all test problems.

Original: While A DAM compares favorably on most test problems, in some cases the other optimizers are considerably better.
Revised: While A DAM compares favorably on most test problems, in some cases the other optimizers are considerably better.

Original: This is most notable on
Revised: This is most notable on

Original: CIFAR-100, where M OMENTUM is signiﬁcantly better then the other two.
Revised: CIFAR-100, where M OMENTUM is signiﬁcantly better then the other two.

Original: The connection between the four learning metrics is non-trivial.
Revised: The connection between the four learning metrics is non-trivial.

Original: Looking at P6 and P7 we note that the optimizers rank differently on train vs. test loss.
Revised: Looking at P6 and P7 we note that the optimizers rank differently on train vs. test loss.

Original: However, there is no optimizer that universally generalizes better than the others; the generalization performance is evidently problem-dependent.
Revised: However, there is no optimizerthat universally generalizes better than the others; the generalization performance is evidently problem-dependent.

Original: The same holds for the generalization from loss to accuracy (e.g. P3 or P6).
Revised: The same holds for the generalization from loss to accuracy (e.g. P3 or P6).

Original: A DAM is somewhat easier to tune.
Revised: A DAM is somewhat easier to tune.

Original: Between the eight test problems, the optimal learning rate for each optimizer varies signiﬁcantly.
Revised: Between the eight test problems, the optimal learning rate for each optimizer varies signiﬁcantly.

Original: Figure 3 shows the ﬁnal performance against learning rate for each of the eight test problems.
Revised: Figure 3 shows the ﬁnal performance against learning rate for each of the eight test problems.

Original: There is no signiﬁcant difference between the three optimizers in terms of their learning rate sensitivity.
Revised: There is no signiﬁcant difference between the three optimizers in terms of their learning rate sensitivity.

Original: However, in most cases, the order of magnitude of the optimal learning rate for A DAM is between 10 − 4 and 10 − 3 (with the exception of P1), while for SGD and 
Revised: However, in most cases, the order of magnitude of the optimal learning rate for A DAM is in the order of 10 − 4 and 10 − 3 (with the exception of P1), while for SGD and M OMENTUM this spread is slightly larger. [Figure]

Original:  M OMENTUM this spread is slightly larger.
Revised: SGD, M OMENTUM , and A DAM produced with D EEP OBS. [Figure]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Deep learning continues to pose a challenging domain for optimization algorithms.
Revised: Deep learning continues to pose a challenging domain for optimization algorithms.

Original: Aspects like stochasticity and generalization make it challenging to benchmark optimization algorithms against each other.
Revised: Aspects like stochasticity and generalization make it challenging to benchmark optimization algorithms against each other.

Original: We have discussed best practices for experimental protocols, and presented the D EEP OBS package, which provide an open-source implementation of these standards.
Revised: We have discussed best practices for experimental protocols, and presented the D EEP OBS package, which provide an open-source implementation of these standards.

Original: We hope that D EEP OBS can help researchers working on optimization for deep learning to build better algorithms, by simultaneously making the empirical evaluation simpler, yet also more reproducible and fair.
Revised: We hope that D EEP OBS can help researchers working on optimization for deep learning to build better algorithms, by simultaneously making the empirical evaluation simpler, yet also more reproducible and fair.

Original: By providing a common ground for methods to be compared on, we aim to speed up the development of deep-learning optimizers, and aid practitioners in their decision for an algorithm.
Revised: By providing a common ground for methods to be compared on, we aim to speed up the development of deep-learning optimizers, and aid practitioners in their decision for an algorithm.

Original: [Figure]
Revised: [Table]

Original: OMENTUM , and A DAM produced with D EEP OBS.
Revised: A CKNOWLEDGMENTS

Original: [Figure] Learning Rate Sensitivity [Figure]
Revised: The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Frank Schneider and Lukas Balles. Lukas Balles and Philipp Hennig gratefully acknowledge support by the ERC action StG 757275 / PANAMA

Original: [Table]
Revised: [Table]

Original: A E XPERIMENTAL SETUP
Revised: A E XPERIMENTAL SETUP

Original: We describe the in total eight test problems that are part of the small and the large benchmark set.
Revised: We describe the in total eight test problems that are part of the small and the large benchmark set.

Original: P1 Noisy Quadratic: A 100 dimensional stochastic quadratic loss function.
Revised: P1 Quadratic Deep: A 100 dimensional stochastic quadratic loss function.

Original: 90% of the eigenvalues are drawn from [0 , 1] , and 10% from [30 , 60] creating an ill-conditioned problem with a structured eigenspectrum similar to the one reported by Chaudhari et al.
Revised: 90% of the eigenvalues are drawn from [0 , 1] , and 10% from [30 , 60] creating an ill-conditioned problem with a structured eigenspectrum similar to the one reported by Chaudhari et al.

Original: We train with a batch size of 128 for 100 epochs.
Revised: We train with a batch size of 128 for 100 epochs.

Original: P2 MNIST — VAE:
Revised: P2 MNIST — VAE:

Original: A variational autoencoder (Kingma & Welling, 2014) with three convolutional and three deconvolutional layers with dropout layers and a latent space of size 8 on the MNIST data set.
Revised: A variational autoencoder (Kingma & Welling, 2014) with three convolutional and three deconvolutional layers with dropout layers and a latent space of size 8 on the MNIST data set.

Original: Trained with a batch size of 64 for 50 epochs.
Revised: Trained with a batch size of 64 for 50 epochs.

Original: P3 F ASHION -MNIST — CNN: A vanilla convolutional network with two convolutional and two fully connected layers for image classiﬁcation on the F ASHION -MNIST data set.
Revised: P3 F ASHION -MNIST — CNN: A vanilla convolutional network with two convolutional and two fully connected layers for image classiﬁcation on the F ASHION -MNIST data set.

Original: Trained with a batch size of 128 for 100 epochs.
Revised: Trained with a batch size of 128 for 100 epochs.

Original: P4 CIFAR-10 — CNN: A slightly larger convolutional network with three convolutional and three fully connected layers on CIFAR-10.
Revised: P4 CIFAR-10 — CNN: A slightly larger convolutional network with three convolutional and three fully connected layers on CIFAR-10.

Original: Trained with a batch size of 128 for 200 epochs.
Revised: Trained with a batch size of 128 for 100 epochs.

Original: P5 F ASHION -MNIST — VAE:
Revised: P5 F ASHION -MNIST — VAE:

Original: A variational autoencoder with three convolutional and three deconvolutional layers with dropout layers and a latent space of size 8 on the F ASHIONMNIST data set.
Revised: A variational autoencoder with three convolutional and three deconvolutional layers with dropout layers and a latent space of size 8 on the F ASHIONMNIST data set.

Original: Trained for 100 epochs with a batch size of 64 .
Revised: Trained for 100 epochs with a batch size of 64 .

Original: P6 S TREET V IEW H OUSE N UMBERS — Wide ResNet-16-4: The wide residual network WRN-16-4 architecture of Zagoruyko & Komodakis (2016) on the S TREET V IEW H OUSE N UMBERS data set for image classiﬁcation.

Original: Trained with a batch size of 128 for 128 epochs.

Original: P7 CIFAR-100 — All-CNN-C:
Revised: P6 CIFAR-100 — All-CNN-C:

Original: The all convolutional network All-CNN-C from Springenberg et al. (2015) for image classiﬁcation on the CIFAR data 
Revised: The all convolutional network All-CNN-C from Springenberg et al. (2015) for image classiﬁcation on the CIFAR-100 data set.

Original: set. Trained with a batch size of 256 for 350 epochs.
Revised: Trained with a batch size of 256 for 350 epochs.

Original: P8 T OLSTOI — CharRNN: A two-layer LSTM (Hochreiter & Schmidhuber, 1997) withunits per LSTM cell for character-level language modeling on T OLSTOI ’s W AR AND P EACE . Trained with a
Revised: P7 S TREET V IEW H OUSE N UMBERS — Wide ResNet-16-4: The wide residual network WRN-16-4 architecture of Zagoruyko & Komodakis (2016) on the S TREET V

Original: sequence length of 50 and batch size

Original: of

Original: for

Original: 200 epochs