Original: H OW MUCH PROGRESS HAVE WE MADE IN NEURAL NETWORK TRAINING ?
Revised: H OW MUCH PROGRESS HAVE WE MADE IN NEURAL NETWORK TRAINING ?

Original: A N EW E VALUATION P ROTO -
Revised: A N EW E VALUATION P ROTO -

Original: COL FOR B ENCHMARKING O PTIMIZERS
Revised: COL FOR B ENCHMARKING O PTIMIZERS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Many optimizers have been proposed for training deep neural networks, and they often have multiple hyperparameters, which make it tricky to benchmark their performance.
Revised: Many optimizers have been proposed for training deep neural networks, and they often have multiple hyperparameters, which make it tricky to benchmark their performance.

Original: In this work, we propose a new benchmarking protocol to evaluate both end-to-end efﬁciency (training a model from scratch without knowing the best hyperparameter) and data-addition training efﬁciency (the previously selected hyperparameters are used for periodically re-training the model with newly collected data).
Revised: In this work, we propose a new benchmarking protocol to evaluate both end-to-end efﬁciency (training a model from scratch without knowing the best hyperparameter) and data-addition training efﬁciency (the previously selected hyperparameters are used for periodically re-training the model with newly collected data).

Original: For end-to-end efﬁciency, unlike previous work that assumes random hyperparameter tuning, which over-emphasizes the tuning time, we propose to evaluate with a bandit hyperparameter tuning strategy.
Revised: For end-to-end efﬁciency, unlike previous work that assumes random hyperparameter tuning, which over-emphasizes the tuning time, we propose to evaluate with a bandit hyperparameter tuning strategy.

Original: A human study is conducted to show that our evaluation protocol matches human tuning behavior better than the random search.
Revised: A human study is conducted to show that our evaluation protocol matches human tuning behavior better than the random search.

Original: For data-addition training, we propose a new protocol for assessing the hyperparameter sensitivity to data shift.
Revised: For data-addition training, we propose a new protocol for assessing the hyperparameter sensitivity to data shift.

Original: We then apply the proposed benchmarking framework to 7 optimizers and various tasks, including computer vision, natural language processing, reinforcement learning, and graph mining.
Revised: We then apply the proposed benchmarking framework to 7 optimizers and various tasks, including computer vision, natural language processing, reinforcement learning, and graph mining.

Original: Our results show that there is no clear winner across all the tasks.
Revised: Our results show that there is no clear winner across all the tasks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Due to the enormous data size and non-convexity, stochastic optimization algorithms have become widely used in training deep neural networks.
Revised: Due to the enormous data size and non-convexity, stochastic optimization algorithms have become widely used in training deep neural networks.

Original: In addition to Stochastic Gradient Descent (SGD) (Robbins & Monro, 1951), many variations such as Adagrad (Duchi et al., 2011) and
Revised: In addition to Stochastic Gradient Descent (SGD) (Robbins & Monro, 1951), many variations such as Adagrad (Duchi et al., 2011) and

Original: Adam (Kingma & Ba, 2014) have been proposed.
Revised: Adam (Kingma & Ba, 2014) have been proposed.

Original: Unlike classical, hyperparameter free optimizers such as gradient descent and Newton’s method 1 , stochastic optimizers often hold multiple hyperparameters including learning rate and momentum coefﬁcients.
Revised: Unlike classical, hyperparameter free optimizers such as gradient descent and Newton’s method 1 , stochastic optimizers often hold multiple hyperparameters including learning rate and momentum coefﬁcients.

Original: Those hyperparameters are critical not only to the speed, but also to the ﬁnal performance, and are often hard to tune.
Revised: Those hyperparameters are critical not only to the speed, but also to the ﬁnal performance, and are often hard to tune.

Original: It is thus non-trivial to benchmark and compare optimizers in deep neural network training.
Revised: It is thus non-trivial to benchmark and compare optimizers in deep neural network training.

Original: And a benchmarking mechanism that focuses on the peak performance could lead to a false sense of improvement when developing new optimizers without considering tuning efforts.
Revised: And a benchmarking mechanism that focuses on the peak performance could lead to a false sense of improvement when developing new optimizers without considering tuning efforts.

Original: In this paper, we aim to rethink the role of hyperparameter tuning in benchmarking optimizers and develop new benchmarking protocols to reﬂect their performance in practical tasks better.
Revised: In this paper, we aim to rethink the role of hyperparameter tuning in benchmarking optimizers and develop new benchmarking protocols to reﬂect their performance in practical tasks better.

Original: We then benchmark seven recently proposed and widely used optimizers and study their performance on a wide range of tasks.
Revised: We then benchmark seven recently proposed and widely used optimizers and study their performance on a wide range of tasks.

Original: In the following, we will ﬁrst brieﬂy review the two existing benchmarking protocols, discuss their pros and cons, and then introduce our contributions.
Revised: In the following, we will ﬁrst brieﬂy review the two existing benchmarking protocols, discuss their pros and cons, and then introduce our contributions.

Original: Benchmarking performance under the best hyperparameters A majority of previous benchmarks and comparisons on optimizers are based on the best hyperparameters.
Revised: Benchmarking performance under the best hyperparameters A majority of previous benchmarks and comparisons on optimizers are based on the best hyperparameters.

Original: Wilson et al.
Revised: Wilson et al.

Original: Shah et al.
Revised: Shah et al.

Original: (2018) made a comparison of SGD-based methods against adaptive ones under their best hyperparameter conﬁgurations.
Revised: (2018) made a comparison of SGD-based methods against adaptive ones under their best hyperparameter conﬁgurations.

Original: They found that SGD can outperform adaptive methods on several datasets under careful tuning.
Revised: They found that SGD can outperform adaptive methods on several datasets under careful tuning.

Original: Most of the benchmarking frameworks for ML training also assume knowing the best hyperparameters for optimizers (Schneider et al., 2019; Coleman et al., 2017; Zhu et al., 2018).
Revised: Most of the benchmarking frameworks for ML training also assume knowing the best hyperparameters for optimizers (Schneider et al., 2019; Coleman et al., 2017; Zhu et al., 2018).

Original: Also, the popular MLPerf benchmark evaluated the performance of optimizers under the best hyperparameter.
Revised: Also, the popular MLPerf benchmark evaluated the performance of optimizers under the best hyperparameter.

Original: It showed that ImageNet and BERT could be trained in 1 minute using the combination of good optimizers, good hyperparameters, and thousands of accelerators.
Revised: It showed that ImageNet and BERT could be trained in 1 minute using the combination of good optimizers, good hyperparameters, and thousands of accelerators.

Original: Despite each optimizer’s peak performance being evaluated, benchmarking under the best hyperparameters makes the comparison between optimizers unreliable and fails to reﬂect their practical performance.
Revised: Despite each optimizer’s peak performance being evaluated, benchmarking under the best hyperparameters makes the comparison between optimizers unreliable and fails to reﬂect their practical performance.

Original: First, the assumption of knowing the best hyperparameter is unrealistic.
Revised: First, the assumption of knowing the best hyperparameter is unrealistic.

Original: In practice, it requires a lot of tuning efforts to ﬁnd the best hyperparameter, and the tuning efﬁciency varies greatly for different optimizers.
Revised: In practice, it requires a lot of tuning efforts to ﬁnd the best hyperparameter, and the tuning efﬁciency varies greatly for different optimizers.

Original: It is also tricky to deﬁne the “best hyperparameter”, which depends on the hyperparameter searching range and grids.
Revised: It is also tricky to deﬁne the “best hyperparameter”, which depends on the hyperparameter searching range and grids.

Original: Further, since many of these optimizers are sensitive to hyperparameters, some improvements reported for new optimizers may come from insufﬁcient tuning for previous work.
Revised: Further, since many of these optimizers are sensitive to hyperparameters, some improvements reported for new optimizers may come from insufﬁcient tuning for previous work.

Original: Benchmarking performance with random hyperparameter search It has been pointed out in several papers that tuning hyperparameter needs to be considered in evaluating optimizers (Schneider et al., 2019; Asi & Duchi, 2019), but having a formal evaluation protocol on this topic is nontrivial.
Revised: Benchmarking performance with random hyperparameter search It has been pointed out in several papers that tuning hyperparameter needs to be considered in evaluating optimizers (Schneider et al., 2019; Asi & Duchi, 2019), but having a formal evaluation protocol on this topic is nontrivial.

Original: Only recently, two papers Choi et al.
Revised: Only recently, two papers Choi et al.

Original: (2019) and Sivaprasad et al.
Revised: (2019) and Sivaprasad et al.

Original: (2020) take hyperparameter tuning time into account when comparing SGD with Adam/Adagrad.
Revised: (2020) take hyperparameter tuning time into account when comparing SGD with Adam/Adagrad.

Original: However, their comparisons among optimizers are conducted on random hyperparameter search.
Revised: However, their comparisons among optimizers are conducted on random hyperparameter search.

Original: We argue that these comparisons could over-emphasize the role of hyperparameter tuning, which could lead to a pessimistic and impractical performance benchmarking for optimizers.
Revised: We argue that these comparisons could over-emphasize the role of hyperparameter tuning, which could lead to a pessimistic and impractical performance benchmarking for optimizers.

Original: This is due to the following reasons:
Revised: This is due to the following reasons:

Original: First, in the random search comparison, each bad hyperparameter has to run fully (e.g., 200 epochs).
Revised: First, in the random search comparison, each bad hyperparameter has to run fully (e.g., 200 epochs).

Original: In practice, a user can always stop the program early for bad hyperparameters if having a limited time budget.
Revised: In practice, a user can always stop the program early for bad hyperparameters if having a limited time budget.

Original: For instance, if the learning rate for SGD is too large, a user can easily observe that
Revised: For instance, if the learning rate for SGD is too large, a user can easily observe that

Original: SGD diverges in a few iterations and directly stops the current job.
Revised: SGD diverges in a few iterations and directly stops the current job.

Original: Therefore, the random search hypothesis will over-emphasize the role of hyperparameter tuning and does not align with a real user’s practical efﬁciency.
Revised: Therefore, the random search hypothesis will over-emphasize the role of hyperparameter tuning and does not align with a real user’s practical efﬁciency.

Original: Second, the performance of the best hyperparameter is crucial for many applications.
Revised: Second, the performance of the best hyperparameter is crucial for many applications.

Original: For example, in many real-world applications, we need to re-train the model every day or every week with newly added data.
Revised: For example, in many real-world applications, we need to re-train the model every day or every week with newly added data.

Original: So the best hyperparameter selected in the beginning might beneﬁt all these re-train tasks rather than searching parameters from scratch.
Revised: So the best hyperparameter selected in the beginning might beneﬁt all these re-train tasks rather than searching parameters from scratch.

Original: In addition, due to the expensive random search, random search based evaluation often focuses on the low-accuracy region 2 , while practically we care about the performance for getting reasonably good accuracy.
Revised: In addition, due to the expensive random search, random search based evaluation often focuses on the low-accuracy region 2 , while practically we care about the performance for getting reasonably good accuracy.

Original: Our contributions Given that hyperparameter tuning is either under-emphasized (assuming the best hyperparameters) or over-emphasize (assuming random search) in existing benchmarking protocols and comparisons, we develop new evaluation protocols to compare optimizers to reﬂect the real use cases better.
Revised: Our contributions Given that hyperparameter tuning is either under-emphasized (assuming the best hyperparameters) or over-emphasize (assuming random search) in existing benchmarking protocols and comparisons, we develop new evaluation protocols to compare optimizers to reﬂect the real use cases better.

Original: Our evaluation framework includes two protocols.
Revised: Our evaluation framework includes two protocols.

Original: First, to evaluate the end-to-end training efﬁciency for a user to train the best model from scratch, we develop an efﬁcient evaluation protocol to compare the accuracy obtained under various time budgets, including the hyperparameter tuning time.
Revised: First, to evaluate the end-to-end training efﬁciency for a user to train the best model from scratch, we develop an efﬁcient evaluation protocol to compare the accuracy obtained under various time budgets, including the hyperparameter tuning time.

Original: Instead of using the random search algorithm, we adopt the Hyperband (Li et al., 2017) algorithm for hyperparameter tuning since it can stop early for bad conﬁgurations and better reﬂect the real running time required by a user.
Revised: Instead of using the random search algorithm, we adopt the Hyperband (Li et al., 2017) algorithm for hyperparameter tuning since it can stop early for bad conﬁgurations and better reﬂect the real running time required by a user.

Original: Further, we also propose to evaluate the data addition training efﬁciency for a user to re-train the model with some newly added training data, with the knowledge of the best hyperparameter tuned in the previous training set.
Revised: Further, we also propose to evaluate the data addition training efﬁciency for a user to re-train the model with some newly added training data, with the knowledge of the best hyperparameter tuned in the previous training set.

Original: We also conduct human studies to study how machine learning researchers are tuning parameters in optimizers and how that aligns with our proposed protocols.
Revised: We also conduct human studies to study how machine learning researchers are tuning parameters in optimizers and how that aligns with our proposed protocols.

Original: Based on the proposed evaluation protocols, we study how much progress has recently proposed algorithms made compared with SGD or Adam .
Revised: Based on the proposed evaluation protocols, we study how much progress has recently proposed algorithms made compared with SGD or Adam .

Original: Note that most of the recent proposed optimizers have shown outperforming SGD and Adam under the best hyperparameters for some particular tasks, but it is not clear whether the improvements are still signiﬁcant when considering hyper-parameter tuning, and across various tasks.
Revised: Note that most of the recent proposed optimizers have shown outperforming SGD and Adam under the best hyperparameters for some particular tasks, but it is not clear whether the improvements are still signiﬁcant when considering hyper-parameter tuning, and across various tasks.

Original: To this end, we conduct comprehensive experiments comparing state-of-the-art training algorithms, including SGD (Robbins & Monro, 1951),
Revised: To this end, we conduct comprehensive experiments comparing state-of-the-art training algorithms, including SGD (Robbins & Monro, 1951),

Original: Adam (Kingma & Ba, 2014), RAdam (Liu et al., 2019), Yogi (Zaheer et al., 2018), LARS (You et al., 2017), LAMB (You et al., 2019), and Lookahead (Zhang et al., 2019), on a variety of training tasks including image classiﬁcation, generated adversarial networks (GANs), sentence classiﬁcation (BERT ﬁne-tuning), reinforcement learning and graph neural network training.
Revised: Adam (Kingma & Ba, 2014), RAdam (Liu et al., 2019), Yogi (Zaheer et al., 2018), LARS (You et al., 2017), LAMB (You et al., 2019), and Lookahead (Zhang et al., 2019), on a variety of training tasks including image classiﬁcation, generated adversarial networks (GANs), sentence classiﬁcation (BERT ﬁne-tuning), reinforcement learning and graph neural network training.

Original: Our main conclusions are: 1) On CIFAR-10 and CIFAR-100, all the optimizers including SGD are competitive.
Revised: Our main conclusions are: 1) On CIFAR-10 and CIFAR-100, all the optimizers including SGD are competitive.

Original: Adaptive methods are generally better on more complex tasks (NLP, GCN, RL).
Revised: Adaptive methods are generally better on more complex tasks (NLP, GCN, RL).

Original: There is no clear winner among adaptive methods.
Revised: There is no clear winner among adaptive methods.

Original: Although RAdam is more stable than Adam across tasks, Adam is still a very competitive baseline even compared with recently proposed methods.
Revised: Although RAdam is more stable than Adam across tasks, Adam is still a very competitive baseline even compared with recently proposed methods.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Optimizers.
Revised: Optimizers.

Original: Properties of deep learning make it natural to apply stochastic
Revised: Properties of deep learning make it natural to apply stochastic

Original: ﬁrst order methods, such as Stochastic Gradient Descent (SGD) (Robbins & Monro, 1951).
Revised: ﬁrst order methods, such as Stochastic Gradient Descent (SGD) (Robbins & Monro, 1951).

Original: Severe issues such as a zig-zag training trajectory and a uniform learning rate have been exposed, and researchers have then drawn extensive attention to modify the existing SGD for improvement.
Revised: Severe issues such as a zig-zag training trajectory and a uniform learning rate have been exposed, and researchers have then drawn extensive attention to modify the existing SGD for improvement.

Original: Along this line of work, tremendous progresses have been made including SGDM (Qian, 1999), Adagrad (Duchi et al., 2011), RMSProp (Tieleman & Hinton, 2012), and Adam (Kingma & Ba, 2014).
Revised: Along this line of work, tremendous progresses have been made including SGDM (Qian, 1999), Adagrad (Duchi et al., 2011), RMSProp (Tieleman & Hinton, 2012), and Adam (Kingma & Ba, 2014).

Original: These methods utilize momentums to stabilize and speed up training procedures.
Revised: These methods utilize momentums to stabilize and speed up training procedures.

Original: Particularly, Adam is regarded as the default algorithm due to its outstanding compatibility.
Revised: Particularly, Adam is regarded as the default algorithm due to its outstanding compatibility.

Original: Then variants such as Amsgrad (Reddi et al., 2019), Adabound (Luo et al., 2019), Yogi (Zaheer et al., 2018), and RAdam (Liu et al., 2019) have been proposed to resolve different drawbacks of Adam.
Revised: Then variants such as Amsgrad (Reddi et al., 2019), Adabound (Luo et al., 2019), Yogi (Zaheer et al., 2018), and RAdam (Liu et al., 2019) have been proposed to resolve different drawbacks of Adam.

Original: Meanwhile, the requirement of large batch training has inspired the development of LARS (You et al., 2017) and LAMB (You et al., 2019).
Revised: Meanwhile, the requirement of large batch training has inspired the development of LARS (You et al., 2017) and LAMB (You et al., 2019).

Original: Moreover, Zhang et al.
Revised: Moreover, Zhang et al.

Original: (2019) has put forward a framework called Lookahead to boost optimization performance by iteratively updating two sets of weights.
Revised: (2019) has put forward a framework called Lookahead to boost optimization performance by iteratively updating two sets of weights.

Original: Hyperparameter tuning methods.
Revised: Hyperparameter tuning methods.

Original: Random search and grid search (Bergstra & Bengio, 2012) can be a basic hyperparameter tuning method in the literature.
Revised: Random search and grid search (Bergstra & Bengio, 2012) can be a basic hyperparameter tuning method in the literature.

Original: However, the inefﬁciency of these methods stimulates the development of more advanced search strategies.
Revised: However, the inefﬁciency of these methods stimulates the development of more advanced search strategies.

Original: Bayesian optimization methods including Bergstra et al.
Revised: Bayesian optimization methods including Bergstra et al.

Original: (2011) and Hutter et al.
Revised: (2011) and Hutter et al.

Original: (2011) accelerate random search by ﬁtting a black-box function of hyperparameter and the expected objective to adaptively guide the search direction.
Revised: (2011) accelerate random search by ﬁtting a black-box function of hyperparameter and the expected objective to adaptively guide the search direction.

Original: Parallel to this line of work, Hyperband (Li et al., 2017) focuses on reducing evaluation cost for each conﬁguration and early terminates relatively worse trials.
Revised: Parallel to this line of work, Hyperband (Li et al., 2017) focuses on reducing evaluation cost for each conﬁguration and early terminates relatively worse trials.

Original: Falkner et al.
Revised: Falkner et al.

Original: (2018) proposes
Revised: (2018) proposes

Original: BOHB to combine the beneﬁts of both Bayesian Optimization and Hyperband.
Revised: BOHB to combine the beneﬁts of both Bayesian Optimization and Hyperband.

Original: All these methods still require huge computation resources.
Revised: All these methods still require huge computation resources.

Original: A recent work (Metz et al., 2020) has tried to obtain a list of potential hyperparameters by meta-learning from thousands of representative tasks.
Revised: A recent work (Metz et al., 2020) has tried to obtain a list of potential hyperparameters by meta-learning from thousands of representative tasks.

Original: We strike a balance between effectiveness and computing cost and leverage Hyperband in our evaluation protocol to compare a wider range of optimizers.
Revised: We strike a balance between effectiveness and computing cost and leverage Hyperband in our evaluation protocol to compare a wider range of optimizers.

Original:  P ROPOSED E VALUATION P ROTOCOLS
Revised:  P ROPOSED E VALUATION P ROTOCOLS

Original: In this section, we introduce the proposed evaluation framework for optimizers.
Revised: In this section, we introduce the proposed evaluation framework for optimizers.

Original: We consider two evaluation protocols, each corresponding to an important training scenario:
Revised: We consider two evaluation protocols, each corresponding to an important training scenario:

Original: • Scenario I
Revised: • Scenario I

Original: (End-to-end training) : This is the general training scenario, where a user is given an unfamiliar optimizer and task, the goal is to achieve the best validation performance after several trials and errors.
Revised: (End-to-end training) : This is the general training scenario, where a user is given an unfamiliar optimizer and task, the goal is to achieve the best validation performance after several trials and errors.

Original: In this case, the evaluation needs to include hyperparameter tuning time.
Revised: In this case, the evaluation needs to include hyperparameter tuning time.

Original: We develop an efﬁciency evaluation protocol to compare various optimizers in terms of CPE and peak performance.
Revised: We develop an efﬁciency evaluation protocol to compare various optimizers in terms of CPE and peak performance.

Original: • Scenario II (Data-addition training) :
Revised: • Scenario II (Data-addition training) :

Original: This is another useful scenario encountered in many applications, where the same model needs to be retrained regularly after collecting some fresh data.
Revised: This is another useful scenario encountered in many applications, where the same model needs to be retrained regularly after collecting some fresh data.

Original: In this case, a naive solution is to reuse the previously optimal hyperparameters and retrain the model.
Revised: In this case, a naive solution is to reuse the previously optimal hyperparameters and retrain the model.

Original: However, since the distribution is shifted, the result depends on the sensitivity to that shift.
Revised: However, since the distribution is shifted, the result depends on the sensitivity to that shift.

Original: We describe the detailed evaluation protocol for each setting in the following subsections.
Revised: We describe the detailed evaluation protocol for each setting in the following subsections.

Original: 3. E ND - TO - END T RAINING E VALUATION P ROTOCOL
Revised: 3. E ND - TO - END T RAINING E VALUATION P ROTOCOL

Original: Before introducing our evaluation protocol for Scenario I, we ﬁrst formally deﬁne the concept of optimizer and its hyperparameters.
Revised: Before introducing our evaluation protocol for Scenario I, we ﬁrst formally deﬁne the concept of optimizer and its hyperparameters.

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: An optimizer is employed to solve a minimization problem min θ L ( θ ) and can be deﬁned by a tuple o ∈ O
Revised: An optimizer is employed to solve a minimization problem min θ L ( θ ) and can be deﬁned by a tuple o ∈ O

Original: =
Revised: =

Original: ( U , Ω) , where O contains all types of optimizers.
Revised: ( U , Ω) , where O contains all types of optimizers.

Original: U is a speciﬁc update rule and Ω = ( ω 1 , . . .
Revised: U is a speciﬁc update rule and Ω = ( ω 1 , . . .

Original: , ω N ) ∈ R N represents a vector of N hyperparameters.
Revised: , ω N ) ∈ R N represents a vector of N hyperparameters.

Original: Search space of these hyperparameters is denoted by F .
Revised: Search space of these hyperparameters is denoted by F .

Original: Given an initial parameter value θ 0 , together with a trajectory of optimization procedure H t = { θ s , L ( θ s ) , ∇L ( θ s ) } , the optimizer updates θ by
Revised: Given an initial parameter value θ 0 , together with a trajectory of optimization procedure H t = { θ s , L ( θ s ) , ∇L ( θ s ) } , the optimizer updates θ by

Original: [Equation]
Revised: [Equation]

Original: U ( H t , Ω) .
Revised: U ( H t , Ω) .

Original: We aim to evaluate the end-to-end time for a user to get the best model, including the hyperparameter tuning time.
Revised: We aim to evaluate the end-to-end time for a user to get the best model, including the hyperparameter tuning time.

Original: A recent work (Sivaprasad et al., 2020) assumes that a user conducts random search for ﬁnding the best hyperparameter setting.
Revised: A recent work (Sivaprasad et al., 2020) assumes that a user conducts random search for ﬁnding the best hyperparameter setting.

Original: Still, we argue that the random search procedure will over-emphasize the importance of hyperparameters when tuning is considered — it assumes a user never stops the training even if they observe divergence or bad results in the initial training phase, which is unrealistic.
Revised: Still, we argue that the random search procedure will over-emphasize the importance of hyperparameters when tuning is considered — it assumes a user never stops the training even if they observe divergence or bad results in the initial training phase, which is unrealistic.

Original: Figure 1 illustrates why random search might not lead to a fair comparison of optimizers.
Revised: Figure 1 illustrates why random search might not lead to a fair comparison of optimizers.

Original: In Figure 1, we are given two optimizers, A and B, and their corresponding loss w.r.t.
Revised: In Figure 1, we are given two optimizers, A and B, and their corresponding loss w.r.t.

Original: hyperparameter.
Revised: hyperparameter.

Original: According to Sivaprasad et al.
Revised: According to Sivaprasad et al.

Original: (2020), optimizer B is considered better than optimizer A under a constrained budget since most regions of the hyperparameter space of A outperforms B.
Revised: (2020), optimizer B is considered better than optimizer A under a constrained budget since most regions of the hyperparameter space of A outperforms B.

Original: For instance, suppose we randomly sample the same hyperparamter setting for A and B.
Revised: For instance, suppose we randomly sample the same hyperparamter setting for A and B.

Original: The ﬁnal conﬁg ω ∗ r ( B ) found under this strategy can have a lower expected loss than that of ω ∗ r ( A ) , as shown in Figure 1a.
Revised: The ﬁnal conﬁg ω ∗ r ( B ) found under this strategy can have a lower expected loss than that of ω ∗ r ( A ) , as shown in Figure 1a.

Original: However, there exists a more practical search strategy which can invalidate this statement with the assumption of a limited searching budget: a user can early terminate a conﬁguration trial when trapped in bad results or diverging.
Revised: However, there exists a more practical search strategy which can invalidate this statement with the assumption of a limited searching budget: a user can early terminate a conﬁguration trial when trapped in bad results or diverging.

Original: Hence, we can observe in Figure 1b that for optimizer A, this strategy earlystops many conﬁgurations and only allow a limited number of trials to explore to the deeper stage.
Revised: Hence, we can observe in Figure 1b that for optimizer A, this strategy earlystops many conﬁgurations and only allow a limited number of trials to explore to the deeper stage.

Original: Therefore, the bad hyperparameters will not affect the overall efﬁciency of Optimizer A too much.
Revised: Therefore, the bad hyperparameters will not affect the overall efﬁciency of Optimizer A too much.

Original: In contrast, for optimizer B, performances of different hyperparameters are relatively satisfactory and hard to distinguish, resulting in similar and long termination time for each trial.
Revised: In contrast, for optimizer B, performances of different hyperparameters are relatively satisfactory and hard to distinguish, resulting in similar and long termination time for each trial.

Original: Therefore, it may be easier for a practical search strategy p to ﬁnd
Revised: Therefore, it may be easier for a practical search strategy p to ﬁnd

Original: the best conﬁguration ω ∗ p ( A ) of optimizer
Revised: the best conﬁguration ω ∗ p ( A ) of optimizer

Original: A than ω ∗ p ( B ) , given the same constrained budget.
Revised: A than ω ∗ p ( B ) , given the same constrained budget.

Original: [Figure]
Revised: [Figure]

Original: A is more sensitive to hyperparamters than optimizers
Revised: A is more sensitive to hyperparamters than optimizers

Original: B, but it may be prefered if bad hyperparameters can be terminated in the early stage.
Revised: B, but it may be prefered if bad hyperparameters can be terminated in the early stage.

Original: This example suggests that random search may over-emphasize the parameter sensitivity when benchmarking optimizers.
Revised: This example suggests that random search may over-emphasize the parameter sensitivity when benchmarking optimizers.

Original: To better reﬂect a practical hyperparameter tuning scenario, our evaluation assumes a user applies Hyperband (Li et al., 2017), a simple but effective hyperparameter tuning scheme to get the best model.
Revised: To better reﬂect a practical hyperparameter tuning scenario, our evaluation assumes a user applies Hyperband (Li et al., 2017), a simple but effective hyperparameter tuning scheme to get the best model.

Original: Hyperband formulates hyperparameter optimization as a unique bandit problem.
Revised: Hyperband formulates hyperparameter optimization as a unique bandit problem.

Original: It accelerates random search through adaptive resource allocation and earlystopping, as demonstrated in Figure 1b.
Revised: It accelerates random search through adaptive resource allocation and earlystopping, as demonstrated in Figure 1b.

Original: Compared with its more complicated counterparts such as
Revised: Compared with its more complicated counterparts such as

Original: BOHB (Falkner et al., 2018), Hyperband requires less computing resources and performs similarly within a constrained budget.
Revised: BOHB (Falkner et al., 2018), Hyperband requires less computing resources and performs similarly within a constrained budget.

Original: The algorithm is presented in Appendix A.
Revised: The algorithm is presented in Appendix A.

Original: [Figure]
Revised: [Figure]

Original: Despite different hyperparameter tuning algorithms, human tuning by experts is still regarded as the most effective.
Revised: Despite different hyperparameter tuning algorithms, human tuning by experts is still regarded as the most effective.

Original: To verify and support that Hyperband is an effective method and even competitive with humans, we conduct a human study as follows: for image classiﬁcation on CIFAR10, given 10 learning rate conﬁgurations of SGD in the grid [1 . 

Revised: To verify and support that Hyperband is an effective method and even competitive with humans, we conduct a human study as follows: for image classiﬁcation on CIFAR10, given 10 learning rate conﬁgurations of SGD in the grid [1 .

Original: × 10 − 8 , 1 . 

Revised: × 10 − 8 , 1 .

Original: × 10 − 7 , 1 . 

Revised: × 10 − 7 , 1 .

Original: × 10 − 6 , . .
Revised: × 10 − 6 , . .

Original: 10] , participants are requested to search the best one at their discretion.
Revised: 10] , participants are requested to search the best one at their discretion.

Original: Namely, they can stop or pause a trial any time and continue to evaluate a new conﬁguration until they feel it has already reached the best performance.
Revised: Namely, they can stop or pause a trial any time and continue to evaluate a new conﬁguration until they feel it has already reached the best performance.

Original: 10 participants are sampled randomly from
Revised: 10 participants are sampled randomly from

Original: Ph.D. students with computer science backgrounds.
Revised: Ph.D. students with computer science backgrounds.

Original: We collect their tuning trajectories and average them as human performance, which is considered “optimal” in this human study.
Revised: We collect their tuning trajectories and average them as human performance, which is considered “optimal” in this human study.

Original: In Figure 2, we plot curves for hyperparameter tuning of human,
Revised: In Figure 2, we plot curves for hyperparameter tuning of human,

Original: Hyperband, random search, random search with an early stopping (ES) strategy in Sivaprasad et al.
Revised: Hyperband, random search, random search with an early stopping (ES) strategy in Sivaprasad et al.

Original: (2020), and Hyperband with ES.
Revised: (2020), and Hyperband with ES.

Original: We ﬁnd that Hyperband matches humans’ behavior better, while random search tends to trap in suboptimal conﬁgurations although random with early stopping can mitigate this issue to some extent.
Revised: We ﬁnd that Hyperband matches humans’ behavior better, while random search tends to trap in suboptimal conﬁgurations although random with early stopping can mitigate this issue to some extent.

Original: This ﬁnding shows the advantage of Hyperband over random search regardless of early stopping, and justiﬁes the use of Hyperband in optimizer benchmarking.
Revised: This ﬁnding shows the advantage of Hyperband over random search regardless of early stopping, and justiﬁes the use of Hyperband in optimizer benchmarking.

Revised: More details of this human study can be found in Appedix B.

Original: With Hyperband incorporated in end-to-end training, we assume that each conﬁguration is run sequentially and record the best performance obtained at time step t as P t .
Revised: With Hyperband incorporated in end-to-end training, we assume that each conﬁguration is run sequentially and record the best performance obtained at time step t as P t .

Original: Speciﬁcally, P t represents the evaluation metric for each task, e.g., accuracy for image classiﬁcation and return for reinforcement learning.
Revised: Speciﬁcally, P t represents the evaluation metric for each task, e.g., accuracy for image classiﬁcation and return for reinforcement learning.

Original: { P t } Tt =1 forms a trajectory for plotting learning curves on test set like Figure 3.
Revised: { P t } Tt =1 forms a trajectory for plotting learning curves on test set like Figure 3.

Original: Although it is intuitive to observe the performance of different optimizers according to such ﬁgures, summarizing a learning curve into a quantiﬁable, scalar value can be more insightful for evaluation.
Revised: Although it is intuitive to observe the performance of different optimizers according to such ﬁgures, summarizing a learning curve into a quantiﬁable, scalar value can be more insightful for evaluation.

Original: Thus, as shown in Eq. 1, we use λ -tunability deﬁned in Sivaprasad et al.
Revised: Thus, as shown in Eq. 1, we use λ -tunability deﬁned in Sivaprasad et al.

Original: (2020) to further measure the performance of optimizers:
Revised: (2020) to further measure the performance of optimizers:

Original: [Equation]
Revised: [Equation]

Original: One intuitive way is to set λ t = 1 t
Revised: One intuitive way is to set λ t = 1 t

Original: = T to determine which optimizer can reach the best model performance after the whole training procedure.
Revised: = T to determine which optimizer can reach the best model performance after the whole training procedure.

Original: However, merely considering the peak performance is not a good guidance on the choice of optimizers.
Revised: However, merely considering the peak performance is not a good guidance on the choice of optimizers.

Original: In practice, we tend to take into account the complete trajectory and exert more emphasis on the early stage.
Revised: In practice, we tend to take into account the complete trajectory and exert more emphasis on the early stage.

Original: Thus, we employ the Cumulative
Revised: Thus, we employ the Cumulative

Original: Performance-Early weighting scheme where λ t ∝ ( T − i ) , to compute λ -tunablity instead of the extreme assignment λ t = 1 t = T .
Revised: Performance-Early weighting scheme where λ t ∝ ( T − i ) , to compute λ -tunablity instead of the extreme assignment λ t = 1 t = T .

Original: The value obtained is termed as CPE for simplicity.
Revised: The value obtained is termed as CPE for simplicity.

Original: We present our evaluation protocol in Algorithm 1.
Revised: We present our evaluation protocol in Algorithm 1.

Original: As we can see, end-to-end training with hyperparameter optimization is conducted for various optimizers on the given task.
Revised: As we can see, end-to-end training with hyperparameter optimization is conducted for various optimizers on the given task.

Original: The trajectory { P t } Tt =1 is recorded to compute the peak performance as well as CPE value.
Revised: The trajectory { P t } Tt =1 is recorded to compute the peak performance as well as CPE value.

Original: Note that the procedure is repeated M times to obtain a reliable result.
Revised: Note that the procedure is repeated M times to obtain a reliable result.

Original: We use M = 3 in all experiments.
Revised: We use M = 3 in all experiments.

Original: More details of time cost and acceleration of the algorithm can be found in Appendix D.
Revised: More details of time cost and acceleration of the algorithm can be found in Appendix E.

Original: 3. D ATA - ADDITION T RAINING E VALUATION P ROTOCOL
Revised: D ATA - ADDITION T RAINING E VALUATION P ROTOCOL

Original: In Scenario II, we have a service (e.g., a search or recommendation engine) and we want to re-train the model every day or every week with some newly added training data.
Revised: In Scenario II, we have a service (e.g., a search or recommendation engine) and we want to re-train the model every day or every week with some newly added training data.

Original: One may argue that an online learning algorithm should be used in this case, but in practice online learning is unstable and industries still prefer this periodically retraining scheme which is more stable.
Revised: One may argue that an online learning algorithm should be used in this case, but in practice online learning is unstable and industries still prefer this periodically retraining scheme which is more stable.

Original: In this scenario, once the best hyperparameters were chosen in the beginning, we can reuse them for every training, so no hyperparameter tuning is required and the performance (including both efﬁciency and test accuracy) under the best hyperparameter becomes important.
Revised: In this scenario, once the best hyperparameters were chosen in the beginning, we can reuse them for every training, so no hyperparameter tuning is required and the performance (including both efﬁciency and test accuracy) under the best hyperparameter becomes important.

Original: However, an implicit assumption made in this process is that “the best hyperparameter will still work when the training task slightly changes” .
Revised: However, an implicit assumption made in this process is that “the best hyperparameter will still work when the training task slightly changes” .

Original: This can be viewed as transferability of hyperparameters for a particular optimizer, and our second evaluation protocol aims to evaluate this practical scenario.
Revised: This can be viewed as transferability of hyperparameters for a particular optimizer, and our second evaluation protocol aims to evaluate this practical scenario.

Original: We simulate data-addition training with all classiﬁcation tasks, and the evaluation protocol works as follows: 1) Extract a subset D δ containing partial training data from the original full dataset D with a small ratio δ ; 2) Conduct a hyperparameter search on D δ to ﬁnd the best setting under this scenario; 3) Use these hyperparameters to train the model on the complete dataset; 4) Observe the potential change of the ranking of various optimizers before and after data addition.
Revised: We simulate data-addition training with all classiﬁcation tasks, and the evaluation protocol works as follows: 1) Extract a subset D δ containing partial training data from the original full dataset D with a small ratio δ ; 2) Conduct a hyperparameter search on D δ to ﬁnd the best setting under this scenario; 3) Use these hyperparameters to train the model on the complete dataset; 4) Observe the potential change of the ranking of various optimizers before and after data addition.

Original: For step 4) when comparing different optimizers, we will plot the training curve in the full-data training stage in Section 4, and also summarize the training curve using the CPE value.
Revised: For step 4) when comparing different optimizers, we will plot the training curve in the full-data training stage in Section 4, and also summarize the training curve using the CPE value.

Original: The detailed evaluation protocol is described in Algorithm 2.
Revised: The detailed evaluation protocol is described in Algorithm 2.

Original: Input optimizero
Revised: optimizer

Original: Hyperband aa partial dataset Dbest hyperparameter setting
Revised: Hyperband a partial dataset D hyperparameter setting

Original: Apply optimizer
Revised: Apply optimizer

Original: D

Original: D

Original: M repetitions
Revised: M repetitions

Original:  E XPERIMENTAL R ESULTS
Revised:  E XPERIMENTAL R ESULTS

Original: Optimizers to be evaluated.
Revised: Optimizers to be evaluated.

Original: As shown in Table 1, we consider 7 optimizers including nonadaptive methods using only the ﬁrst-order momentum, and adaptive methods considering both ﬁrstorder and second-order momentum.
Revised: As shown in Table 1, we consider 7 optimizers including nonadaptive methods using only the ﬁrst-order momentum, and adaptive methods considering both ﬁrstorder and second-order momentum.

Original: We also provide lists of tunable hyperparameters for different optimizers in Table 1.
Revised: We also provide lists of tunable hyperparameters for different optimizers in Table 1.

Original: Moreover, we consider following two combinations of tunable hyperparameters to better investigate the performance of different optimizers: a) only tuning initial learning rate with the others set to default values and b) tuning a full list of hyperparameters.
Revised: Moreover, we consider following two combinations of tunable hyperparameters to better investigate the performance of different optimizers: a) only tuning initial learning rate with the others set to default values and b) tuning a full list of hyperparameters.

Original: A detailed description of optimizers as well as default values and search range of these hyperparameters can be found in Appendix D. Note that we adopt a uniﬁed search space for a fair comparison following Metz et al.
Revised: A detailed description of optimizers as well as default values and search range of these hyperparameters can be found in Appendix E. Note that we adopt a uniﬁed search space for a fair comparison following Metz et al.

Original: to eliminate biases of speciﬁc ranges for different optimizers.
Revised: to eliminate biases of speciﬁc ranges for different optimizers.

Original: The tuning budget of
Revised: The tuning budget of

Original: Hyperband is determined by three items: maximum resource (in this paper we use epoch) per conﬁguration R , reduction factor η , and number of conﬁgurations n c .
Revised: Hyperband is determined by three items: maximum resource (in this paper we use epoch) per conﬁguration R , reduction factor η , and number of conﬁgurations n c .

Original: According to Li et al.
Revised: According to Li et al.

Original: (2017), a single Hyperband execution contains n s = (cid:98) log η ( R ) (cid:99) + 1 of SuccessiveHalving, each referred to as a bracket.
Revised: (2017), a single Hyperband execution contains n s = (cid:98) log η ( R ) (cid:99) + 1 of SuccessiveHalving, each referred to as a bracket.

Original: These brackets take strategies from least to most aggressive early-stopping, and each one is designed to use approximately B = R · n s resources, leading to a ﬁnite total budget.
Revised: These brackets take strategies from least to most aggressive early-stopping, and each one is designed to use approximately B = R · n s resources, leading to a ﬁnite total budget.

Original: The number of randomly sampled conﬁgurations in one Hyperband run is also ﬁxed and grows with R .
Revised: The number of randomly sampled conﬁgurations in one Hyperband run is also ﬁxed and grows with R .

Original: Then given R and η , n c determines the repetition times of Hyperband.
Revised: Then given R and η , n c determines the repetition times of Hyperband.

Original: We set η = 3 as this default value performs consistently well, and R to a value which each task usually takes for a complete run.
Revised: We set η = 3 as this default value performs consistently well, and R to a value which each task usually takes for a complete run.

Original: For n c , it is assigned as what is required for a single Hyperband execution for all tasks, except for
Revised: For n c , it is assigned as what is required for a single Hyperband execution for all tasks, except for

Original: BERT ﬁne-tuning, where a larger number of conﬁgurations is necessary due to a relatively small R .
Revised: BERT ﬁne-tuning, where a larger number of conﬁgurations is necessary due to a relatively small R .

Original: In Appendix D, we give assigned values of R , η , and n c for each task.
Revised: In Appendix E, we give assigned values of R , η , and n c for each task.

Original: [Table]
Revised: [Table]

Original: Tasks for benchmarking.
Revised: Tasks for benchmarking.

Original: For a comprehensive and reliable assessment of optimizers, we consider a wide range of tasks in different domains.
Revised: For a comprehensive and reliable assessment of optimizers, we consider a wide range of tasks in different domains.

Original: When evaluating end-to-end training efﬁciency, we implement our protocol on tasks covering several popular and promising applications in Table 2.
Revised: When evaluating end-to-end training efﬁciency, we implement our protocol on tasks covering several popular and promising applications in Table 2.

Original: Apart from common tasks in computer vision and natural language processing, we introduce two extra tasks in graph neural network training and reinforcement learning.
Revised: Apart from common tasks in computer vision and natural language processing, we introduce two extra tasks in graph neural network training and reinforcement learning.

Original: For simplicity, we will use the dataset to represent each task in our subsequent tables of experimental results.
Revised: For simplicity, we will use the dataset to represent each task in our subsequent tables of experimental results.

Original: (For the reinforcement learning task, we just use the environment name.)
Revised: (For the reinforcement learning task, we just use the environment name.)

Original: The detailed settings and parameters for each task can be found in Appendix C.
Revised: The detailed settings and parameters for each task can be found in Appendix D.

Original: [Table]
Revised: [Table]

Original: 4. E ND - TO - END EFFICIENCY (S ECNARIO I)
Revised: 4. E ND - TO - END EFFICIENCY (S ECNARIO I)

Original: To evaluate end-to-end training efﬁciency, we adopt the protocol in Algorithm 1.
Revised: To evaluate end-to-end training efﬁciency, we adopt the protocol in Algorithm 1.

Original: Speciﬁcally, we record the average training trajectory with Hyperband { P t } Tt =1 for each optimizer on benchmarking tasks, where P t is the evaluation metric for each task (e.g., accuracy, return).
Revised: Speciﬁcally, we record the average training trajectory with Hyperband { P t } Tt =1 for each optimizer on benchmarking tasks, where P t is the evaluation metric for each task (e.g., accuracy, return).

Original: We visualize these trajectories in Figure 3 for CIFAR10 and CIFAR100, and calculate CPE and peak performance in
Revised: We visualize these trajectories in Figure 3 for CIFAR10 and CIFAR100, and calculate CPE and peak performance in

Original: Table 3 and 9 respectively.
Revised: Table 3 and 9 respectively.

Original: More results for other tasks and the peak performance can be found in Appendix E. Besides, in Eq.
Revised: More results for other tasks and the peak performance can be found in Appendix F. Besides, in Eq.

Original: 2 we compute performance ratio r o,a for each optimizer and each task, and then utilize the distribution function of a performance metric called performance proﬁle
Revised: 2 we compute performance ratio r o,a for each optimizer and each task, and then utilize the distribution function of a performance metric called performance proﬁle

Original: ρ o ( τ ) to summarize the performance of different optimizers over all the tasks.
Revised: ρ o ( τ ) to summarize the performance of different optimizers over all the tasks.

Original: For tasks where a lower CPE is better, we just use r o,a = CPE o,a / min { CPE o,a } instead to guarantee r o,a ≥ 1 .
Revised: For tasks where a lower CPE is better, we just use r o,a = CPE o,a / min { CPE o,a } instead to guarantee r o,a ≥ 1 .

Original: The function ρ o ( τ ) for all optimizers is presented in Figure 4.
Revised: The function ρ o ( τ ) for all optimizers is presented in Figure 4.

Original: Based on the deﬁnition of performance proﬁle (Dolan & Mor´e, 2002), the optimizers with large probability ρ o ( τ ) are to be preferred.
Revised: Based on the deﬁnition of performance proﬁle (Dolan & Mor´e, 2002), the optimizers with large probability ρ o ( τ ) are to be preferred.

Original: In particular, the value of ρ o (1) is the probability that one optimizer will win over the rest and can be a reference for selecting the proper optimizer for an unknown task.
Revised: In particular, the value of ρ o (1) is the probability that one optimizer will win over the rest and can be a reference for selecting the proper optimizer for an unknown task.

Original: We also provided a probabilistic performance proﬁle to summarize different optimizers in Figure 7 in Appendix E.
Revised: We also provided a probabilistic performance proﬁle to summarize different optimizers in Figure 7 in Appendix F.

Original: [Equation]
Revised: [Equation]

Original: Our ﬁndings are summarized below
Revised: Our ﬁndings are summarized below

Original: : • It should be emphasized from Table 3 and 9, that under our protocol based on Hyperband, SGD performs similarly to Adam in terms of efﬁciency as well as peak performance, and can even surpass it in some cases like training on CIFAR100.
Revised: : • It should be emphasized from Table 3 and 9, that under our protocol based on Hyperband, SGD performs similarly to Adam in terms of efﬁciency as well as peak performance, and can even surpass it in some cases like training on CIFAR100.

Original: Under Hyperband, the best conﬁguration of SGD is less tedious to ﬁnd than random search because Hyperband can early-stop bad runs and thus they will affect less to the search efﬁciency and ﬁnal performance.
Revised: Under Hyperband, the best conﬁguration of SGD is less tedious to ﬁnd than random search because Hyperband can early-stop bad runs and thus they will affect less to the search efﬁciency and ﬁnal performance.

Original: • For image classiﬁcation tasks all the methods are competitive, while adaptive methods tend to perform better in more complicated tasks (NLP, GCN, RL).
Revised: • For image classiﬁcation tasks all the methods are competitive, while adaptive methods tend to perform better in more complicated tasks (NLP, GCN, RL).

Original: • There is no signiﬁcant distinction among adaptive variants.
Revised: • There is no signiﬁcant distinction among adaptive variants.

Original: Performance of adaptive optimizers tends to fall in the range within 1% of the best result.
Revised: Performance of adaptive optimizers tends to fall in the range within 1% of the best result.

Original: • According to performance proﬁle in Figure 4, the RAdam achieves probability 1 with the smallest τ , and Adam is the second method achieving that.
Revised: • According to performance proﬁle in Figure 4, the RAdam achieves probability 1 with the smallest τ , and Adam is the second method achieving that.

Original: This indicates that RAdam and Adam are achieving relatively stable and consistent performance among these tasks.
Revised: This indicates that RAdam and Adam are achieving relatively stable and consistent performance among these tasks.

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: 4. D ATA - ADDITION T RAINING (S CENARIO II)
Revised: 4. D ATA - ADDITION T RAINING (S CENARIO II)

Original: We then conduct evaluation on data-addition training based on the protocol in Algorithm 2.
Revised: We then conduct evaluation on data-addition training based on the protocol in Algorithm 2.

Original: We choose four classiﬁcation problems on CIFAR10, CIFAR100, MRPC and PPI since the setting do not apply to RL.
Revised: We choose four classiﬁcation problems on CIFAR10, CIFAR100, MRPC and PPI since the setting do not apply to RL.

Original: We search the best hyperparameter conﬁguration, denoted by Ω partial , under the sub training set with the ratio δ = 0 . 3 .
Revised: We search the best hyperparameter conﬁguration, denoted by Ω partial , under the sub training set with the ratio δ = 0 . 3 .

Original: Here we tune all hyperparameters.
Revised: Here we tune all hyperparameters.

Original: Then we directly apply Ω partial on the full dataset for a complete training process.
Revised: Then we directly apply Ω partial on the full dataset for a complete training process.

Original: The training curves are shown in Figure
Revised: The training curves are shown in Figure

Original: [Table]
Revised: [Table]

Original: and we also summarize the training curve with CPE by Eq.
Revised: and we also summarize the training curve with CPE by Eq.

Original: 1 in Table 4.
Revised: 1 in Table 4.

Original: We have the following ﬁndings:
Revised: We have the following ﬁndings:

Original: • There is no clear winner in data-addition training.
Revised: • There is no clear winner in data-addition training.

Original: RAdam is outperforming other optimizers in 2/ tasks so is slightly preferred, but other optimizers except Lookahead are also competitive (within 1% range) on at least 2/4 tasks.
Revised: RAdam is outperforming other optimizers in 2/ tasks so is slightly preferred, but other optimizers except Lookahead are also competitive (within 1% range) on at least 2/4 tasks.

Original: • To investigate whether the optimizer’s ranking will change when adding 70% data, we compare the training curve on the original 30% data versus the training curve on the full 100% data in Figure 5.
Revised: • To investigate whether the optimizer’s ranking will change when adding 70% data, we compare the training curve on the original 30% data versus the training curve on the full 100% data in Figure 5.

Original: We observe that the ranking of optimizers slightly changes after data addition.
Revised: We observe that the ranking of optimizers slightly changes after data addition.

Original: [Table]
Revised: [Table]

Original:  C ONCLUSIONS AND D ISCUSSIONS
Revised:  C ONCLUSIONS AND D ISCUSSIONS

Original: In conclusion, we found there is no strong evidence that newly proposed optimizers consistently outperform Adam , while each of them may be good for some particular tasks.
Revised: In conclusion, we found there is no strong evidence that newly proposed optimizers consistently outperform Adam , while each of them may be good for some particular tasks.

Original: When deciding the choice of the optimizer for a speciﬁc task, people can refer to results in Table 3 and 9.
Revised: When deciding the choice of the optimizer for a speciﬁc task, people can refer to results in Table 3 and 9.

Original: If the task is contained in Table 2, he/she can directly choose the one with the best CPE or best peak performance based on his/her goal of the task (easy to tune or high ﬁnal performance).
Revised: If the task is contained in Table 2, he/she can directly choose the one with the best CPE or best peak performance based on his/her goal of the task (easy to tune or high ﬁnal performance).

Original: On the other hand, even though the desired task is covered, people can also gain some insights from the results of the most similar task in Table 2, or refer to the performance proﬁle in Figure 4 to pick adaptive methods like Adam.
Revised: On the other hand, even though the desired task is covered, people can also gain some insights from the results of the most similar task in Table 2, or refer to the performance proﬁle in Figure 4 to pick adaptive methods like Adam.

Original: Besides choosing the optimizer, it will contribute to designing a new optimizer as well.
Revised: Besides choosing the optimizer, it will contribute to designing a new optimizer as well.

Original: Using our protocol to evaluate a new optimizer can show whether it has obvious improvement over existing optimizers, and can serve as a routine to judge the performance of the optimizer thoroughly.
Revised: Using our protocol to evaluate a new optimizer can show whether it has obvious improvement over existing optimizers, and can serve as a routine to judge the performance of the optimizer thoroughly.

Original: [Figure]
Revised: [Figure]

Original: In addition to the proposed two evaluation criteria, there could be other factors that affect the practical performance of an optimizer.
Revised: In addition to the proposed two evaluation criteria, there could be other factors that affect the practical performance of an optimizer.

Original: First, the memory consumption is becoming important for training large DNN models.
Revised: First, the memory consumption is becoming important for training large DNN models.

Original: For instance, although Lookahead performs well in certain tasks, it requires more memory than other optimizers, restricting their practical use in some memory constrained applications.
Revised: For instance, although Lookahead performs well in certain tasks, it requires more memory than other optimizers, restricting their practical use in some memory constrained applications.

Original: Another important criterion is the scalability of optimizers.
Revised: Another important criterion is the scalability of optimizers.

Original: When training with a massively distributed system, optimizing the performance of a large batch regime (e.g., 32K batch size for ImageNet) is important.
Revised: When training with a massively distributed system, optimizing the performance of a large batch regime (e.g., 32K batch size for ImageNet) is important.

Original: LARS and LAMB algorithms included in our study are developed for large batch training.
Revised: LARS and LAMB algorithms included in our study are developed for large batch training.

Original: We believe it is another important metric for comparing optimizers worth studying further.
Revised: We believe it is another important metric for comparing optimizers worth studying further.

Original: A H YPER B AND
Revised: A H YPER B AND

Original: We present the whole algorithm for Hyperband in Algorithm 3, and you can refer to Li et al.
Revised: We present the whole algorithm for Hyperband in Algorithm 3, and you can refer to Li et al.

Original: (2017) for more details.
Revised: (2017) for more details.

Original: Input: R, η
Revised: Input: R, η

Original: Initialization: s max = (cid:98) log
Revised: Initialization: s max = (cid:98) log

Original: η R (cid:99) , B = ( s max + 1) R 1: for s ∈ { s max , s max − 1 , . . .
Revised: η R (cid:99) , B = ( s max + 1) R 1: for s ∈ { s max , s max − 1 , . . .

Original: , 0 } do 2: n = (cid:100)
Revised: , 0 } do 2: n = (cid:100)

Original: BR η s ( s +1) (cid:101) , r = Rη − s 3: // begin SuccessiveHalving with ( n, r ) inner loop 4:
Revised: BR η s ( s +1) (cid:101) , r = Rη − s 3: // begin SuccessiveHalving with ( n, r ) inner loop 4:

Original: T = random get conﬁguration ( n ) 5: for i ∈ { 0 , . . .
Revised: T = random get conﬁguration ( n ) 5: for i ∈ { 0 , . . .

Original: , s } do 6: n i = (cid:98)
Revised: , s } do 6: n i = (cid:98)

Original: nη − i (cid:99) , r i = rη i 7:
Revised: nη − i (cid:99) , r i = rη i 7:

Original: L = { run then return val loss ( t, r i ) : t ∈ T } 8:
Revised: L = { run then return val loss ( t, r i ) : t ∈ T } 8:

Original: T = top k
Revised: T = top k

Original: ( T, L, (cid:98) n i /η (cid:99) ) 9: end for 10: end for return Hyperparameter conﬁguration with the smallest loss seen so far
Revised: ( T, L, (cid:98) n i /η (cid:99) ) 9: end for 10: end for return Hyperparameter conﬁguration with the smallest loss seen so far

Original: B O PTIMIZERS
Revised: B D ETAILS OF H UMAN S TUDY

Revised: In this human study, 10 participants are sampled from Ph.D. students with computer science backgrounds (machine learning backgrounds speciﬁcally). They are recruited as follows: We ﬁrst asked the administrators to distribute the program of this human study to Ph.D. students in machine learning labs in our institutions.

Revised: Provided the population, we assumed that they had prior knowledge about some basic machine learning experiments, such as image classiﬁcation on MNIST and CIFAR10.

Revised: They were requested to conduct hyperparameter tuning of learning rate based on their knowledge.

Original: {+They were informed that the target experiment was image classiﬁcation on CIFAR with SGD, and the search range of learning rate was in the grid [1 . 
+}
Revised: {+They were informed that the target experiment was image classiﬁcation on CIFAR with SGD, and the search range of learning rate was in the grid [1 . 
+}

Original: {+× 10 − 8 , 1 . 
+}
Revised: {+× 10 − 8 , 1 . 
+}

Original: {+× 10 − 7 , 1 . 
+}
Revised: {+× 10 − 7 , 1 . 
+}

Revised: × 10 − 6 , . .

Revised: Each conﬁguration was run for 200 epochs at most.

Revised: Moreover, we also told them that they could pause any conﬁguration if they wanted to evaluate others, and even stop the whole tuning process only when they thought the accuracy could not be further improved and the number of total tuning epochs exceeded 600 at the same time.

Revised: We collected results from 17 people and we determined the validity by checking whether the length of the trajectory was greater than 600 and removed 7 invalidity trajectories.

Revised: Finally there remained 10 trajectories and we averaged them as the human performance in Figure 2.

Revised: C O PTIMIZERS

Original: Notations.
Revised: Notations.

Original: Given a vector of parameters θ ∈ R d , we denote a sub-vector of its i-th layer’s parameters by θ ( i ) .
Revised: Given a vector of parameters θ ∈ R d , we denote a sub-vector of its i-th layer’s parameters by θ ( i ) .

Original: { α t } Tt =1 is a sequence of learning rates during the optimization procedure of a horizon
Revised: { α t } Tt =1 is a sequence of learning rates during the optimization procedure of a horizon

Original: T .
Revised: T .

Original: { φ t , ψ t } Tt =1 represents a sequence of functions to calculate the ﬁrst-order and second-order momentum of the gradient g t , which are m t and v t respectively at time step t .
Revised: { φ t , ψ t } Tt =1 represents a sequence of functions to calculate the ﬁrst-order and second-order momentum of the gradient g t , which are m t and v t respectively at time step t .

Original: Different optimization algorithms are usually speciﬁed by the choice of φ ( · ) and ψ ( · ) .
Revised: Different optimization algorithms are usually speciﬁed by the choice of φ ( · ) and ψ ( · ) .

Original: { r t } Tt =1 is an additional sequence of adaptive terms to modify the magnitude of the learning rate in some methods.
Revised: { r t } Tt =1 is an additional sequence of adaptive terms to modify the magnitude of the learning rate in some methods.

Original: For algorithms only using the ﬁrst-order momentum, µ is the , while β 1 and β 2 are coefﬁcients to compute the running averages m and v .
Revised: For algorithms only using the ﬁrst-order momentum, µ is the , while β 1 and β 2 are coefﬁcients to compute the running averages m and v .

Original: (cid:15) is a small scalar (e.g., 1 × 10 − 8 ) used to prevent division by 0 .
Revised: (cid:15) is a small scalar (e.g., 1 × 10 − 8 ) used to prevent division by 0 .

Original: Generic optimization framework.
Revised: Generic optimization framework.

Original: Based on, we further develop a thorough generic optimization framework including an extra adaptive term in Algorithm 4.
Revised: Based on, we further develop a thorough generic optimization framework including an extra adaptive term in Algorithm 4.

Original: The debiasing term used in the original version of Adam is ignored for simplicity.
Revised: The debiasing term used in the original version of Adam is ignored for simplicity.

Original: Note that for { α t } T t =1 , different learning rate scheduling strategies can be adopted and the choice of scheduler is also regarded as a tunable hyperparameter.
Revised: Note that for { α t } T t =1 , different learning rate scheduling strategies can be adopted and the choice of scheduler is also regarded as a tunable hyperparameter.

Original: Without loss of generality, in this paper, we only consider a constant value and a linear decay (Shallue et al., 2018) in the following equation, introducing γ as a hyperparameter.
Revised: Without loss of generality, in this paper, we only consider a constant value and a linear decay (Shallue et al., 2018) in the following equation, introducing γ as a hyperparameter.

Original: [Equation]
Revised: [Equation]

Original: With this generic framework, we can summarize several popular optimization methods by explicitly specifying m t , v t and r t in Table 5.
Revised: With this generic framework, we can summarize several popular optimization methods by explicitly specifying m t , v t and r t in Table 5.

Original: It should be clariﬁed that Lookahead is an exception of the generic framework.
Revised: It should be clariﬁed that Lookahead is an exception of the generic framework.

Original: In fact it is more like a high-level mechanism, which can be incorporated with any other optimizer.
Revised: In fact it is more like a high-level mechanism, which can be incorporated with any other optimizer.

Original: However, as stated in Zhang et al.
Revised: However, as stated in Zhang et al.

Original: (2019), this optimizer is robust to inner optimization algorithm, k , and α s in Algorithm 5, we still include Lookahead here with Adam as the base for a more convincing and comprehensive evaluation.
Revised: (2019), this optimizer is robust to inner optimization algorithm, k , and α s in Algorithm 5, we still include Lookahead here with Adam as the base for a more convincing and comprehensive evaluation.

Original: We consider Lookahead as a special adaptive method, and tune the same hyperparamters for it as other adaptive optimziers.
Revised: We consider Lookahead as a special adaptive method, and tune the same hyperparamters for it as other adaptive optimziers.

Original: Algorithm Input: parameter value θ 1 , learning rate with scheduling { α t } , sequence of functions { φ t , ψ t , χ t } Tt =1 to compute m t , v t , and r t respectively.
Revised: Input: parameter value θ 1 , learning rate with scheduling { α t } , sequence of functions { φ t , ψ t , χ t } Tt =1 to compute m t , v t , and r t respectively.

Original: 1: for t = 1 to T do 2: g t = ∇ f t ( θ t ) 3: m t =
Revised: 1: for t = 1 to T do 2: g t = ∇ f t ( θ t ) 3: m [Equation]

Original: φ t ( g 1 , · · · , g t ) 4: v t = ψ t ( g 1 , · · · , g t ) 5: r t = χ t ( θ t , m t , v t ) 6: θ t +1 =
Revised: , g [Equation] [Equation] [Equation]

Original: θ t − α t r t m t / √ v t 7: end for
Revised: [Equation] [Equation] v t 7: end for

Original: [Table]
Revised: [Table]

Original: v t − 1 + (1 − β 2 ) g 2 t RAdam β 1 m t − 1 + (1 − β 1 ) g t β 2 v t − 1 + (1 − β 2 ) g 2 t (cid:113) ( ρ t − 4)( ρ t − 2) ρ ∞ ( ρ ∞ − 4)( ρ ∞ − 2) ρ t Yogi β 1 m t − 1 + (1 − β 1 ) g t v t − 1 − (1 − β 2 ) sign( v t − 1 − g 2 t ) g 2 t LARS µm t − 1 + g t (cid:107) θ
Revised: [Table]

Original: ( i ) t (cid:107) / (cid:107) m
Revised: [Equation]

Original: ( i ) t (cid:107)
Revised: [Table]

Original: LAMB β 1 m t − 1 + (1 − β 1 ) g t β 2 v t − 1 + (1 − β 2 ) g 2 t (cid:107) θ ( i ) t (cid:107) (cid:107) m ( i ) t / (cid:113) v ( i ) t (cid:107) Lookahead* β 1 m t − 1 + (1 − β 1 ) g
Revised: [Equation]

Original: t β 2 v t − 1 + (1 − β 2 ) g 2 t
Revised: [Table] [Equation]

Original: Input: Initial parameters θ 0 , objective function f , synchronization period k , slow weights step size α s , optimizer A 1: for t = 1 , 2 , . . .
Revised: Input: Initial parameters θ 0 , objective function f , synchronization period k , slow weights step size α s , optimizer A 1: for t = 1 , 2 , . . .

Original: do 2:
Revised: do 2:

Original: Synchronize parameters ˆ θ t, 0 ← θ t −3: for i = 1 , 2 , . . .
Revised: Synchronize parameters ˆ θ t, 0 ← θ t −3: for i = 1 , 2 , . . .

Original: , k do 4:
Revised: , k do 4:

Original: Sample minibatch of data d ∈ D 5:
Revised: Sample minibatch of data d ∈ D 5:

Original: ˆ θ t,i ← ˆ θ t,i − 1 + A ( L, ˆ θ t,i − 1 , d ) 6: end for 7:
Revised: ˆ θ t,i ← ˆ θ t,i − 1 + A ( L, ˆ θ t,i − 1 , d ) 6: end for 7:

Original: Perform outer update θ t ← θ t − 1 + α s (ˆ θ t,k − θ t − 1 ) 8: end for return Parameters θ
Revised: Perform outer update θ t ← θ t − 1 + α s (ˆ θ t,k − θ t − 1 ) 8: end for return Parameters θ

Original: C T ASK DESCRIPTION
Revised: D T ASK DESCRIPTION

Original: We make a concrete description of tasks selected for our optimizer evaluation protocol:
Revised: We make a concrete description of tasks selected for our optimizer evaluation protocol:

Original: • Image classifcation.
Revised: • Image classifcation.

Original: For this task, we adopt a ResNet-

Revised: For this task, we adopt a ResNet-

Original: (He et al., 2016) model on CIFAR10 and CIFAR100 with a batch size of 128 and the maximum epoch of 200 per trial.
Revised: (He et al., 2016) model on CIFAR10 and CIFAR100 with a batch size of 128 and the maximum epoch of 200 per trial.

Original: • VAE.
Revised: • VAE.

Original: We use a vanilla variational autoencoder in Kingma & Welling (2013) with ﬁve convolutional and ﬁve deconvolutional layers with a latent space of dimension 128 on CelebA. There are no dropout layers.
Revised: We use a vanilla variational autoencoder in Kingma & Welling (2013) with ﬁve convolutional and ﬁve deconvolutional layers with a latent space of dimension 128 on CelebA. There are no dropout layers.

Original: Trained with a batch size of 144 . • GAN.
Revised: Trained with a batch size of 144 . • GAN.

Original: We train SNGAN with the same network architecture and objective function with spectral normalization for CIFAR10 in Miyato et al.
Revised: We train SNGAN with the same network architecture and objective function with spectral normalization for CIFAR10 in Miyato et al.

Original: (2018), and the batch size of the generator and the discriminator is 128 and 64 respectively.
Revised: (2018), and the batch size of the generator and the discriminator is 128 and 64 respectively.

Original: • Natural language processing.
Revised: • Natural language processing.

Original: In this domain, we ﬁnetune RoBERTa-base on MRPC, one of the test suit in GLUE benchmark.
Revised: In this domain, we ﬁnetune RoBERTa-base on MRPC, one of the test suit in GLUE benchmark.

Original: For each optimizer, we set the maximal exploration budget to be 800 epochs.
Revised: For each optimizer, we set the maximal exploration budget to be 800 epochs.

Original: The batch size is 16 sentences.
Revised: The batch size is 16 sentences.

Original: • Graph learning.
Revised: • Graph learning.

Original: Among various graph learning problems, we choose node classiﬁcation as semisupervised classiﬁcation.
Revised: Among various graph learning problems, we choose node classiﬁcation as semisupervised classiﬁcation.

Original: In GCN training, in there are multiple ways to deal with the neighborhood explosion of stochastic optimizers.
Revised: In GCN training, in there are multiple ways to deal with the neighborhood explosion of stochastic optimizers.

Original: We choose Cluster-GCN Chiang et al.
Revised: We choose Cluster-GCN Chiang et al.

Original: as the backbone to handle neighborhood expansion and PPI as the dataset.
Revised: as the backbone to handle neighborhood expansion and PPI as the dataset.

Original: • Reinforcement learning.
Revised: • Reinforcement learning.

Original: We select Walker2d-v3 from OpenAI Gym (Brockman et al.
Revised: We select Walker2d-v3 from OpenAI Gym (Brockman et al.

Original: as our training environment, and PPO (Schulman et al.
Revised: as our training environment, and PPO (Schulman et al.

Original: (2017)), implemented by OpenAI SpinningUp (Achiam (2018)), as the algorithm that required tuning.
Revised: (2017)), implemented by OpenAI SpinningUp (Achiam (2018)), as the algorithm that required tuning.

Original: We use the same architectures for both action value network Q and the policy network π .
Revised: We use the same architectures for both action value network Q and the policy network π .

Original: We deﬁne 40,000 of environment interactions as one epoch, with a batch size of 4,000.
Revised: We deﬁne 40,000 of environment interactions as one epoch, with a batch size of 4,000.

Original: The return we used is the highest average test return of an epoch during the training.
Revised: The return we used is the highest average test return of an epoch during the training.

Original: D I MPLEMENTATION D ETAILS
Revised: E I MPLEMENTATION D ETAILS

Original: Implementation details of our experiments are provided in this section.
Revised: Implementation details of our experiments are provided in this section.

Original: Speciﬁcally, we give the uniﬁed search space for all hyperparamters and their default values in Table 6.
Revised: Speciﬁcally, we give the uniﬁed search space for all hyperparamters and their default values in Table 6.

Original: Note that we tune the learning rate decay factor for image classiﬁcation tasks when tuning every hyperparamter.
Revised: Note that we tune the learning rate decay factor for image classiﬁcation tasks when tuning every hyperparamter.

Original: For the task on MRPC, γ is tuned for all experiments.
Revised: For the task on MRPC, γ is tuned for all experiments.

Original: In other cases, we only tune original hyperparamters without a learning rate scheduler.
Revised: In other cases, we only tune original hyperparamters without a learning rate scheduler.

Original: In addition, Hyperband parameter values for each task are listed in Table 7.
Revised: In addition, Hyperband parameter values for each task are listed in Table 7.

Original: These parameters are assigned based on properties of different tasks.
Revised: These parameters are assigned based on properties of different tasks.

Original: [Table]

Original: [Table]

Original: For time cost of our evaluation protocols, it depends on how many budgets are available.
Revised: For time cost of our evaluation protocols, it depends on how many budgets are available.

Original: Speciﬁcally, in our paper, the unit of time budget is one epoch, then the total time will be B epoch ∗ T epoch , where B epoch is the total available budget and T epoch is the running time for one epoch.
Revised: Speciﬁcally, in our paper, the unit of time budget is one epoch, then the total time will be B epoch ∗ T epoch , where B epoch is the total available budget and T epoch is the running time for one epoch.

Original: There is no additional computational cost, i.e., running our protocol once takes the same time as running one hyperparameter search with Hyperband.
Revised: There is no additional computational cost, i.e., running our protocol once takes the same time as running one hyperparameter search with Hyperband.

Original: In our experiment on CIFAR10, we roughly evaluatedhyperparameter conﬁgurations in one Hyperband running, while the same time can only allow about 50 conﬁgurations in random search. 
Revised: In our experiment on CIFAR10, we roughly evaluatedhyperparameter conﬁgurations in one Hyperband running, while the same time can only allow about 50 conﬁgurations in random search. [Table] SGD [Table]

Original:  Moreover, we can further accelerate our evaluation protocol by resampling, shown in Algorithm 6.
Revised: LARS [Table] [Table] Moreover, we can further accelerate our evaluation protocol by resampling, shown in Algorithm 6.

Original: The basic idea is that we keep a library of different hyperparameter settings.
Revised: The basic idea is that we keep a library of different hyperparameter settings.

Original: At the beginning, the library is empty.
Revised: At the beginning, the library is empty.

Original: And in each repetition, we sample a number of conﬁgurations required by running
Revised: And in each repetition, we sample a number of conﬁgurations required by running

Original: Hyperband once.
Revised: Hyperband once.

Original: During the simulation of Hyperband, we just retrieve the value from the library if the desired epoch of current conﬁguration is contained in the library.
Revised: During the simulation of Hyperband, we just retrieve the value from the library if the desired epoch of current conﬁguration is contained in the library.

Original: Otherwise, we run this conﬁguration based on Hyperband, and store the piece of the trajectory to the library. 
Revised: Otherwise, we run this conﬁguration based on Hyperband, and store the piece of the trajectory to the library. Algorithm

Original: Input: A set of optimizers O = { o : o =
Revised: Input: A set of optimizers O = { o : o =

Original: ( U , Ω) } , task a ∈ A , search space F , library size S 1: for o ∈
Revised: ( U , Ω) } , task a ∈ A , search space F , library size S 1: for o ∈

Original: O do 2:
Revised: O do 2:

Original: Sample S conﬁgurations from F , initialize the library with an empty list for each setting 3: for i = 1 to M do 4:
Revised: Sample S conﬁgurations from F , initialize the library with an empty list for each setting 3: for i = 1 to M do 4:

Original: Simulate Hyperband with o using conﬁgurations re-sampled from the library on a 5: if the desired accuracy is pre-computed in the library then 6:
Revised: Simulate Hyperband with o using conﬁgurations re-sampled from the library on a 5: if the desired accuracy is pre-computed in the library then 6:

Original: Retrieve the value directly 7: else 8:
Revised: Retrieve the value directly 7: else 8:

Original: Training normally, and store values to the library 9: end if 10:
Revised: Training normally, and store values to the library 9: end if 10:

Original: Record the performance trajectory { P t } Tt =1 explored by HyperBand 11:
Revised: Record the performance trajectory { P t } Tt =1 explored by HyperBand 11:

Original: Calculate the peak performance and CPE by Eq.12: end for 13:
Revised: Calculate the peak performance and CPE by Eq.12: end for 13:

Original: Average peak and CPE values over M repetitions for the optimizer o 14: end for 15: Evaluate optimizers according to their peak and CPE values
Revised: Average peak and CPE values over M repetitions for the optimizer o 14: end for 15: Evaluate optimizers according to their peak and CPE values

Original: E A DDITIONAL RESULTS
Revised: F A DDITIONAL RESULTS

Original: More detailed experimental results are reported in this section.
Revised: More detailed experimental results are reported in this section.

Original: E. I MPACT OF η
Revised: F. I MPACT OF η

Original: Since there is an extra hyperparameter, the reduction factor η in Hyperband, we conduct an experiment with different values ( η = 2 , 3 , 4 , 5) to observe the potential impact of this additional hyperparameter on our evaluation.
Revised: Since there is an extra hyperparameter, the reduction factor η in Hyperband, we conduct an experiment with different values ( η = 2 , 3 , 4 , 5) to observe the potential impact of this additional hyperparameter on our evaluation.

Original: Speciﬁcally, we use Hyperband to tune the learning rate for three optimizers, SGD, Adam, and Lookahead on CIFAR10, and results are presented in Table 8.
Revised: Speciﬁcally, we use Hyperband to tune the learning rate for three optimizers, SGD, Adam, and Lookahead on CIFAR10, and results are presented in Table 8.

Original: As we can see, although the change of η may lead to different CPE values, the relative ranking among three optimizers remains unchanged.
Revised: As we can see, although the change of η may lead to different CPE values, the relative ranking among three optimizers remains unchanged.

Original: Besides, they all achieve comparable peak performance at the end of training.
Revised: Besides, they all achieve comparable peak performance at the end of training.

Original: Considering the efﬁciency of Hyperband, we choose η = 3 based on the convention in
Revised: Considering the efﬁciency of Hyperband, we choose η = 3 based on the convention in

Original: Li et al.
Revised: Li et al.

Original: (2017) in all our experiments.
Revised: (2017) in all our experiments.

Original: [Table]
Revised: [Table]

Original: E. E ND - TO - END TRAINING
Revised: F. E ND - TO - END TRAINING

Original: Table 9 shows peak performance for optimizers on each task.
Revised: Table 9 shows peak performance for optimizers on each task.

Original: For GAN, we only conduct evaluation on optimizers tuning learning rate due to time limit, and present its CPE and peak performance in
Revised: For GAN, we only conduct evaluation on optimizers tuning learning rate due to time limit, and present its CPE and peak performance in

Original: Table 10.
Revised: Table 10.

Original: There is also an end-to-end training curve for GAN on CIFAR10 in Figure 6.
Revised: There is also an end-to-end training curve for GAN on CIFAR10 in Figure 6.

Original: Figures for end-to-end training curves on the rest of tasks are shown in Figure 9 and10.
Revised: Figures for end-to-end training curves on the rest of tasks are shown in Figure 9 and

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Adam 77.04 (25.14) RAdam 73.85 (19.61)
Revised: [Table]

Original: Yogi 76.80 (25.36)
Revised: [Table] [Table] [Table]

Original: LARS 157.71 (73.82)
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: Lookahead 65.61 ( 20.40 )
Revised: Besides the general performance proﬁle

Original: [Figure]
Revised: [Figure]

Original: Besides the general performance proﬁle, we present a probabilistic version described in Barreto et al.
Revised: we present a probabilistic version described in Barreto et al.

Original: (2010) in Figure 7 to account for randomness.
Revised: (2010) in Figure 7 to account for randomness.

Original: The probabilistic performance proﬁle can be formulated as
Revised: The probabilistic performance proﬁle can be formulated as

Original: [Equation]
Revised: [Equation]

Original: where µ o,a and σ o,a are the mean and standard deviation of CPE of the optimizer o on the task a respetively
Revised: where µ o,a and σ o [Figure] a are the mean and standard deviation of CPE of the optimizer o on the task a respetively

Original: [Figure]
Revised: [Figure]

Original: and b a is just the best expectation of CPE on a among all optimizers.
Revised: and b a is just the best expectation of CPE on a among all optimizers.

Original: It can be seen in
Revised: It can be seen in

Original: Figure 7 that the probabilistic performance proﬁles shows a similar trend to Figure 
Revised: Figure 7 that the probabilistic performance proﬁles shows a similar trend to Figure 4.

Original: [Figure]
Revised: [Figure]

Original: We also attach two end-to-endtraining trajectories on CIFAR10 with the error in Figure 8.
Revised: We also attach two end-to-end trajectories on CIFAR10 with the error in Figure 8.

Original: Since it is hard to distinguish optimizers with the standard deviation added, we just instead report the std of
Revised: Since it is hard to distinguish optimizers with the standard deviation added, we just instead report the std of

Original: CPE and peak performance in Table 3 and
Revised: CPE and peak performance in Table 3 and

Original: [Figure]
Revised: [Figure]

Original: y

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: E. D ATA ADDITION TRAINING
Revised: D ATA ADDITION TRAINING

Original: We provide training curves for data-addition training on full MRPC and PPI dataset in Figure 11.
Revised: We provide training curves for data-addition training on full MRPC and PPI dataset in Figure 11.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure][Figure]
Revised: [Figure] [Figure]