Original: Reproducibility Report SAdam: A Variant of Adam for Strongly Convex Functions
Revised: Reproducibility Report SAdam: A Variant of Adam for Strongly Convex Functions

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AfﬁliationAddress email

Original: Reproducibility Summary
Revised: Reproducibility Summary

Original: Scope of Reproducibility
Revised: Scope of Reproducibility

Original: The central claim of the original paper is that a modiﬁcation of Adam to utilize strong convexity yields faster convergenceowing to a data dependent logarithmic regret bound, much akin to the recently proposed SC-RMSProp.
Revised: The central claim of the original paper is that a modiﬁcation of Adam to utilize strong convexity yields faster convergence owing to a data dependent logarithmic regret bound, much akin to the recently proposed SC-RMSProp.

Original: Additionallythe authors also claim that the higher performance is sustained even in non-convex optimization problems.
Revised: Additionally the authors also claim that the higher performance is sustained even in non-convex optimization problems.

Original: We runthe same experiments as the ones listed in the original paper, trying to reproduce the observations using a differentframework (PyTorch) as compared to the authors (TensorFlow)[1].
Revised: We run the same experiments as the ones listed in the original paper, trying to reproduce the observations using a differentframework (PyTorch) as compared to the authors (TensorFlow)

Original: Methodology
Revised: Methodology

Original: We did not use any of the source code provided, preferring to implement everything from the scratch, however we didrun the source code to verify the results, using the code provided itself.
Revised: We did not use any of the source code provided, preferring to implement everything from the scratch, however we didrun the source code to verify the results, using the code provided itself.

Original: Although we did vary the hyperparameters tosee the difference on the results obtained, the  values mentioned in the original paper, proved to work the best.
Revised: Additionally we performed a grid search overall the hyperparameters to get the optimal values of the same for each of the datasets.

Original: We usedthe NVIDIA Tesla K80{+K
+} GPU provided on the Google Colab platform to perform all our experiments.
Revised: We used the NVIDIA Tesla {+K
+} GPU provided on the Google Colab platform to perform all our experiments.

Original: Results
Revised: Results

Original: All the claims of the paper, are in line with our empirical observations as well, and our results are within a range of 1-2% of accuracy.
Revised: All the claims of the paper, are in line with our empirical observations as well, and our results are within a range of 1-2% of accuracy.

Original: For one of the experiment listed in the appendix of the original paper involving training of ResNet-18with all of the optimizers, our results deviate.
Revised: For one of the experiment listed in the appendix of the original paper involving training of ResNet-18with all of the optimizers, our results deviate.

Original: However, the same does not discredit any of the claims made by theoriginal paper, as that additional experiment merely was shown to prove an additional beneﬁt of using SAdam in theoriginal paper.
Revised: However, the same does not discredit any of the claims made by theoriginal paper, as that additional experiment merely was shown to prove an additional beneﬁt of using SAdam in theoriginal paper.

Original: What was easy
Revised: What was easy

Original: All experiments were easy to setup, from procuring all the 3 datasets to the selection of the hyperparameters.
Revised: All experiments were easy to setup, from procuring all the 3 datasets to the selection of the hyperparameters.

Original: In addition,none of the experiments had a high runtime requirement, hence no additional hardware was required to conduct theexperiments and training on Google Colab was sufﬁcient.
Revised: In addition,none of the experiments had a high runtime requirement, hence no additional hardware was required to conduct theexperiments and training on Google Colab was sufﬁcient.

Original: What was difﬁcult
Revised: What was difﬁcult

Original: Implementing the optimizers against which SAdam was to be compared took up a lot of time and compounded the workat hand.
Revised: Implementing the optimizers against which SAdam was to be compared took up a lot of time and compounded the workat hand.

Original: Besides that a few 
Revised: Besides that a few contradictions in the proposed values

Original: Communication with original authors
Revised: Communication with original authors

Original: correspondence

Original:  Introduction
Revised:  Introduction

Original: Any variation with respect to Adam with a solid mathematical backing deserves a serious look and hence SAdam [2] gotour attention.
Revised: Any variation with respect to Adam with a solid mathematical backing deserves a serious look and hence SAdam [1] gotour attention.

Original: SAdam is an online convex optimizer that enhances the Adam algorithm by utilizing strong convexity offunctions wherever possible.
Revised: SAdam is an online convex optimizer that enhances the Adam algorithm by utilizing strong convexity offunctions wherever possible.

Original: Although the motivation behind making these modiﬁcations are to improve performancein only convex cases, they prove to be effective even in non-convex cases.
Revised: Although the motivation behind making these modiﬁcations are to improve performancein only convex cases, they prove to be effective even in non-convex cases.

Original: The Online Convex Optimization (OCO)framework involves iterative improvement of the model over a sequence of rounds.
Revised: The Online Convex Optimization (OCO)framework involves iterative improvement of the model over a sequence of rounds.

Original: The OCO framework models thedecision set as a convex set in Euclidean space denoted K ⊂ R n .
Revised: The OCO framework models thedecision set as a convex set in Euclidean space denoted K ⊂ R n .

Original: In each round t the player (our model) chooses adecision x t from K .
Revised: In each round t the player (our model) chooses adecision x t from K .

Original: At the same time, the adversary reveals a cost function f t ( . ) and the player incurs a cost of f t ( x t ) .
Revised: At the same time, the adversary reveals a cost function f t ( . ) and the player incurs a cost of f t ( x t ) .

Original: The difference between the cost incurred by the player and the cost incurred for taking the best decision is called Regret,and the minimization of this value entails improvement of the model.
Revised: The difference between the cost incurred by the player and the cost incurred for taking the best decision is called Regret,and the minimization of this value entails improvement of the model.

Original: [Equation]
Revised: [Equation]

Original: Intuitively, an algorithm performs well if its regret is sublinear as a function of T where T is the time horizon i.e.
Revised: Intuitively, an algorithm performs well if its regret is sublinear as a function of T where T is the time horizon i.e.

Original: R Algo ( T ) =
Revised: R Algo ( T ) =

Original: o ( T ) , and as the complexity of the regret decision decreases, the performance of the optimization increases.
Revised: o ( T ) , and as the complexity of the regret decision decreases, the performance of the optimization increases.

Original: Adam dynamically adjusts the step size and the update direction by exponential average of the past gradients.
Revised: Adam dynamically adjusts the step size and the update direction by exponential average of the past gradients.

Original: Although
Revised: Although

Original: Adam has been extremely successful in many applications, it suffers from the non-convergence issue.
Revised: Adam has been extremely successful in many applications, it suffers from the non-convergence issue.

Original: AMSgrad and
Revised: AMSgrad and

Original: AdamNC were introduced to address these issues.
Revised: AdamNC were introduced to address these issues.

Original: These variants boast a data-dependant regret bounds that are O ( √ T )in the worst case.
Revised: These variants boast a data-dependant regret bounds that are O ( √ T )in the worst case.

Original: SAdam is the ﬁrst variant of Adam adapted to strongly convex functions.
Revised: SAdam is the ﬁrst variant of Adam adapted to strongly convex functions.

Original: It follows the general framework of Adam,yet keeping a faster decaying rate step size controlled by time-variant hyperparameters to exploit strong convexity.
Revised: It follows the general framework of Adam,yet keeping a faster decaying rate step size controlled by time-variant hyperparameters to exploit strong convexity.

Original: Although it must be mentioned that similar ideas have successfully been applied in the past to the frameworks of
Revised: Although it must be mentioned that similar ideas have successfully been applied in the past to the frameworks of

Original: Adagrad and RMSProp to get the variants
Revised: Adagrad and RMSProp to get the variants

Original: called
Revised: called

Original: SC- Adagrad and SC-RMSProp [3].
Revised: SC-Adagrad and SC-RMSProp [2].

Original: Theoretical analysis of SAdamshow a data-dependent O ( logT ) regret bound for strongly convex functions, which means that it converges faster than
Revised: Theoretical analysis of SAdamshow a data-dependent O ( logT ) regret bound for strongly convex functions, which means that it converges faster than

Original: AMSgrad and AdamNC in such cases.
Revised: AMSgrad and AdamNC in such cases.

Original:  Scope of reproducibility
Revised:  Scope of reproducibility

Original: We extrapolate the following claims from the original paper -
Revised: We extrapolate the following claims from the original paper -

Original: • Modiﬁcation of Adam (henceforth referred to as SAdam) to utilize strong convexity yields faster convergenceowing to a data dependent logarithmic regret bound, much akin to the recently proposed SC-RMSProp
Revised: • Modiﬁcation of Adam (henceforth referred to as SAdam) to utilize strong convexity yields faster convergenceowing to a data dependent logarithmic regret bound, much akin to the recently proposed SC-RMSProp

Original: • SAdam ends up outperforming all the existing optimization techniques in the context of deeper network
Revised: • SAdam ends up outperforming all the existing optimization techniques in the context of deeper network

Original: training as well, which is inherently a non-convex optimization problem.
Revised: training as well, which is inherently a non-convex optimization problem.

Original: • A speciﬁc conﬁguration of hyperparameters in SAdam generates a data-dependant logarithmic regret boundfor SC-RMSprop algorithm.
Revised: • A speciﬁc conﬁguration of hyperparameters in SAdam generates a data-dependant logarithmic regret boundfor SC-RMSprop algorithm.

Original: We seek to verify  theﬁrst 2 claims made by the authors empirically as done by the authors themselves, as the otherclaim is theoretical, and the base paper provides sufﬁcient proofs to uphold the veritibility of the same.
Revised: We seek to verify all the claims listed above empirically replicating the experiments of the authors. We skip over thetheoritical nuances of all these claims since the base paper provides detailed analysis replete with self-sufﬁcient proofs.

Original:  Methodology
Revised:  Methodology

Original: The authors made their source code available to us, upon request (now available on the OpenReview Portal as well).
Revised: The authors made their source code available to us, upon request (now available on the OpenReview Portal as well).

Original: Their experimental setup made use of the Tensorﬂow framework, whilst utilizing the publicly available implementations[4] of the other optimizers against which comparisons were made.
Revised: Their experimental setup made use of the Tensorﬂow framework, whilst utilizing the publicly available implementations[3] of the other optimizers against which comparisons were made.

Original: However we implemented the codebase from scratchusing the PyTorch 1.7 framework, which meant reimplementing the other optimizers in question along with SAdam,due to a lack of a publicly available implementation of the same in PyTorch.
Revised: However we implemented the codebase from scratchusing the PyTorch 1.7 framework, which meant reimplementing the other optimizers in question along with SAdam,due to a lack of a publicly available implementation of the same in PyTorch.

Original: All experiments were carried out on
Revised: All experiments were carried out on

Original: Google Colab with the NVIDIA Tesla T4.
Revised: Google Colab with the NVIDIA Tesla K80.

Original: The following experiments were conducted -
Revised: The following experiments were conducted -

Original: • Calculation of Regret for L2 Regularized Logistic Regression on MNIST, CIFAR10, CIFAR100 for our poolof optimizers, to check the validity of claim 1.
Revised: • Calculation of Regret for L2 Regularized Logistic Regression on MNIST, CIFAR10, CIFAR100 for our poolof optimizers, to check the validity of claim 1.

Original: • Computing test accuracy and training loss for a 4-layer CNN and ResNet18 on MNIST, CIFAR10, CIFAR100.for our pool of optimizers 
Revised: • Computing test accuracy and training loss for a 4-layer CNN and ResNet18 on MNIST, CIFAR10, CIFAR100.for our pool of optimizers to check the validity of claim 2.

Original:  to check the validity of claim 2.
Revised: • Training a Multi-Layer LSTM on the PennTreeBank dataset on the Language Modelling task, to test the performance of SAdam in a context different from the usual vision domains it has hitherto been tested on

Original: 3.1 Model Description & Experimental Setup
Revised: 3.1 Model Description & Experimental Setup

Original: All of the optimizer algorithms try to learn the parameter vectors { w i , b i } Ki =1 , by minimizing the loss accumulated overeach mini-batch of training samples { ( x i , y i ) } mi =1 , where y i ∈
Revised: All of the optimizer algorithms try to learn the parameter vectors { w i , b i } Ki =1 , by minimizing the loss accumulated overeach mini-batch of training samples { ( x i , y i ) } mi =1 , where y i ∈

Original: [ K ] , ∀ i ∈ [ m ] , in each round t .
Revised: [ K ] , ∀ i ∈ [ m ] , in each round t .

Original: This loss function alsocalled the Cross Entropy Loss [5] has been used across all our experiments.
Revised: This loss function alsocalled the Cross Entropy Loss [4] has been used across all our experiments.

Original: [Equation]
Revised: [Equation]

Original: 3.1.1 L2 Regularised Softmax Regression
Revised: 3.1.1 L2 Regularised Softmax Regression

Original: A sequential layer is added and the outputs of the same are directly fed into the loss function.
Revised: A sequential layer is added and the outputs of the same are directly fed into the loss function.

Original: It must be noted thatthe PyTorch implementation of Categorical Cross Entropy Loss inherently applies Softmax, while computing the loss,hence to leverage their implementation we remove the last softmax function from our model, and directly feed in thefully connected layers output to the Loss function.
Revised: It must be noted thatthe PyTorch implementation of Categorical Cross Entropy Loss inherently applies Softmax, while computing the loss,hence to leverage their implementation we remove the last softmax function from our model, and directly feed in thefully connected layers output to the Loss function.

Original: We report the regret for this machine learning model across all ourdatasets and optimizers, considering the expert model of choice, to be the one trained by the Adam optimizer.
Revised: We report the regret for this machine learning model across all ourdatasets and optimizers, considering the expert model of choice, to be the one trained by the Adam optimizer.

Original: Thischoice is completely trivial, and a different choice of an expert model would lead to a quantitative change of results, butwould have no effect on the inferences obtained from the experiments.
Revised: Thischoice is completely trivial, and a different choice of an expert model would lead to a quantitative change of results, butwould have no effect on the inferences obtained from the experiments.

Original: 3.1.2 4 Layer CNN
Revised: 3.1.2 4 Layer CNN

Original: To verify the second claim, pertaining to the efﬁcacy of SAdam in non-convex cases, we test the performance of theoptimizers on a custom 4 layer CNN, consisting of two convolutional layers (each with 32 ﬁlters of size 3 X 3 ), onemax-pooling layer (with a 2 X 2 window and 0 .
Revised: To verify the second claim, pertaining to the efﬁcacy of SAdam in non-convex cases, we test the performance of theoptimizers on a custom 4 layer CNN, consisting of two convolutional layers (each with 32 ﬁlters of size 3 X 3 ), onemax-pooling layer (with a 2 X 2 window and 0 .

Original: 25 dropout), and one fully connected layer (with 128 hidden units and0 .
Revised: 25 dropout), and one fully connected layer (with 128 hidden units and0 .

Original: 5 dropout).
Revised: 5 dropout).

Original: We employ ReLU function as the activation function for convolutional layers and softmax function as theactivation function for the fully connected ones.
Revised: We employ ReLU function as the activation function for convolutional layers and softmax function as theactivation function for the fully connected ones.

Original: We train this network on all of our datasets, using all of the optimizers.
Revised: We train this network on all of our datasets, using all of the optimizers.

Original: 3.1.3 ResNet 18
Revised: 3.1.3 ResNet 18

Original: To further substantiate the claim of the authors, we choose a deeper usage of architecture to see, whether the higherlevels of performance are sustained as the number of layers increase.
Revised: To further substantiate the claim of the authors, we choose a deeper usage of architecture to see, whether the higherlevels of performance are sustained as the number of layers increase.

Original: The popular ResNet-18 architecture was used forthis purpose, the details of which can be found in the base paper [6].
Revised: The popular ResNet-18 architecture was used forthis purpose, the details of which can be found in the base paper [5].

Original: To verify the claims made by the authors, weconduct the same experiment as them, running our models 10 times and ﬁnally taking a mean of the aggregate resultson the CIFAR 10 dataset. 
Revised: To verify the claims made by the authors, weconduct the same experiment as them, running our models 10 times and ﬁnally taking a mean of the aggregate resultson the CIFAR 10 dataset. 3.1.4 Multi-Layer LSTM

Revised: To test the performance of SAdam in unchartered territories, since it has hitherto only been used in vision problems, weused SAdam to train a language model on the PennTree bank dataset.

Original: 3.2 Datasets
Revised: Both a regularised and non-regularised RNNmodel containing 2 layers of LSTM was trained. For the regularised version, 650 hidden units were used in each layerof the LSTM, with dropout set at 0.5 and the model was trained for 39 epochs, whilst for the non-regularised model,the number of hidden units chosen was 200 and the model was trained for 13 epochs. In both cases, the LSTMs areunrolled over 35 steps, and trained with a batch size of 20 , while for both vaildation and testing the batch size is set as10 . A learning rate scheduler is also used to aid in training, which decays the learning rate by a factor of 1 . 2 after every6 steps.

Revised: The model conﬁguration and the experimental setup has been used as a basis for assessing the performance of anew optimizer as can be seen in the paper for AdaBelief.[6]

Revised: 3.2 Datasets

Original: The following datasets were used across experiments -
Revised: The following datasets were used across experiments -

Original: • MNIST :
Revised: • MNIST :

Original: Dataset containing 60000 28X28 training images and 10000 test images depicting handwritten digitsspread evenly across each digit (0-9).
Revised: Dataset containing 60000 28X28 training images and 10000 test images depicting handwritten digitsspread evenly across each digit (0-9).

Original: [7]• CIFAR10 :
Revised: • CIFAR10 :

Original: A subset of the 80 million tiny images dataset and consists of 60,000 32x32 color images containingone of 10 object classes, with 6000 images per class.
Revised: A subset of the 80 million tiny images dataset and consists of 60,000 32x32 color images containing one of 10 object classes, with 6000 images per class.

Original: It was collected by Alex Krizhevsky, Vinod Nair, andGeoffrey Hinton.[8]• CIFAR100 :
Revised: It was collected by Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton.[8]

Original:  Subset of the same dataset as CIFAR10, containing 600 images (500 for training and 100 fortesting) per class for 100 classes.
Revised: • CIFAR100 : Subset of the same dataset as CIFAR10, containing 600 images (500 for training and 100 fortesting) per class for 100 classes.

Original: The 100 classes in the CIFAR-100 are grouped into 20 superclasses.[8]
Revised: The 100 classes in the CIFAR-100 are grouped into 20 superclasses.[8]

Revised: PennTreeBank : The Penn Tree Bank dataset maintained by the University of Pennsylvania contains textspanning around 2400 Wall Street Journal material along with a fully tagged Brown Corpus, and has over4 million annotated words over a 10000 worded vocabulary.

Revised: For the language model task, a 929K wordedtraining set, 73k worded validation set, and 82k worded test is provided.

Original: 3.3 Hyperparameters
Revised: 3.3 Hyperparameters

Original: An exhaustive list of hyperparameter settings used across optimizers and experiments have been surmised in table 1.
Revised: An exhaustive list of hyperparameter settings used across optimizers and experiments have been surmised in table 1.

Original: Inthe table α refers to the learning rate to be used, t refers to the step number, and thus in all convex cases, the learningrate is replaced by a degrading learning rate to account for the convergence.
Revised: Inthe table α refers to the learning rate to be used, t refers to the step number, and thus in all convex cases, the learningrate is replaced by a degrading learning rate to account for the convergence.

Original: The β 1 and β 2 parameters appears in Adamand all of its variants controls the gradient moment decay rates as mentioned in [9].
Revised: The β 1 and β 2 parameters appears in Adamand all of its variants controls the gradient moment decay rates as mentioned in [10].

Original: The λ 1 and λ 2 parameters controlthe regularization in the cross entropy loss function, and is used only in 
Revised: The λ 1 and λ 2 parameters controlthe regularization in the cross entropy loss function, and is used only in convex cases.

Revised: δ refers to a scalar value, thathelps in avoiding divide by zero errors in optimizer updates.

Revised: However, in the SC variants of RMSProp and Adagrad atime variant δ t,i is used with the hyperparameters ξ 1 and ξ 2 controlling the variance.

Revised: For SAdam, the optimal values forsome of the hyperparameters mentioned in table 1 differ from experiment to experiment, and have been explored deeplyin the subsequent section.

Original: [Table]
Revised: [Table]

Revised: 3.3.1 Hyperparameter Search

Revised: We use a grid search to determine parameters that provides the best performance for the algorithms in the regret computation experiment.

Revised: We choose β 1 from { 0 . 0 , 0 . 9 } .

Revised: Taking β 2 = 1 − γt we choose γ from { 0 . 9 , 0 . 99 , 0 . 999 , 0 . 9999 } .

Revised: The learning rate α is chosen from { 0 . 1 , 0 . 01 , 0 .

Revised: These are taken as these were the values used by Kingmaet al.

Revised: [10] while testing the original Adam optimizer and used by some subsequent derivative papers as well [11].

Revised: The

Revised: Figures below show the regret values for each parameter conﬁguration averaged over 5 random seeds, individually forour vision datasets.

Revised: Dataset

Revised: [Table]

Revised: As reported in the table above, the best hyperparameter conﬁguration per experiment differs from the correspondingvalues reported in the base paper.

Revised: It must be noted though that the results for the optimal parameters are not overwhelmingly better than the values recommended.

Revised: Fine tuning the hyperparameters in some of the experiments mightlead to slightly better results, but the recommended values of 0 . 9 and 0 .

Revised: 9 for β 1 and γ respectively, irrespective of theexperimental setup would lead to results closer to the best values possibly attainable using SAdam as an optimizer.

Original: 3.4 Computational requirements
Revised: 3.4 Computational requirements

Original: All the experiments listed in the paper were carried out on the Google Colab platform, with the hardware accelerationprovided by NVIDIA Tesla K80 GPU.
Revised: All the experiments listed in the paper were carried out on the Google Colab platform, with the hardware accelerationprovided by NVIDIA Tesla K80 GPU.

Original: As none of the experiments took more than 3 hours to completion, no cost had tobe incurred on our part for conducting any of the experiments.
Revised: As none of the experiments took more than 3 hours to completion, no cost had tobe incurred on our part for conducting any of the experiments.

Original: The list of average runtimes across experiments NVIDIA
Revised: The list of average runtimes across experiments NVIDIA

Original: Tesla K80 GPU can be seen in the table below -
Revised: Tesla K80 GPU can be seen in the table belowModel Dataset

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: CIFAR

Original: [Table]
Revised: [Table]

Revised: Non-Regularised

Original: The calculation of regret in the case of softmax regression warrants the availability of an expert model.
Revised: [Table] LSTM PennTreeBank Regularised

Original:  Thus, themajority of the runtime is spent in obtaining this model, which once ready enables quicker completion of the regretcalculations, since in that case, all of the other optimizers will only have to be trained over 1 epoch.
Revised: [Table] PennTreeBank The calculation of regret in the case of softmax regression warrants the availability of an expert model. Thus, themajority of the runtime is spent in obtaining this model, which once ready enables quicker completion of the regretcalculations, since in that case, all of the other optimizers will only have to be trained over 1 epoch.

Original: After the trainedmodel is avaiable, calculation of regret on a CPU takes an average of 3 - 4 minutes per experiment, and if the same isdone over a GPU, the experiment would take around 40 - 50 seconds on an average.
Revised: After the trainedmodel is avaiable, calculation of regret on a CPU takes an average of 3 - 4 minutes per experiment, and if the same isdone over a GPU, the experiment would take around 40 - 50 seconds on an average.

Original:  Results
Revised:  Results

Original: All the claims of the paper, are in line with our empirical observations as well.
Revised: All the claims of the paper, are in line with our empirical observations as well.

Original: The original paper also compared theperformance of SAdam to that of AdamNC, however we skip that particular comparison, as AMSGrad already servesas an extension for the Adam algorithm to improve upon the convergence of its optimization process, and the inclusionof AdamNC does not enrich the comparisons with any new information, with its performance almost being a replica of
Revised: The original paper also compared theperformance of SAdam to that of AdamNC, however we skip that particular comparison, as AMSGrad already servesas an extension for the Adam algorithm to improve upon the convergence of its optimization process, and the inclusionof AdamNC does not enrich the comparisons with any new information, with its performance almost being a replica of

Original: AMSGrad and Adam.
Revised: AMSGrad and Adam.

Original: Aside from that fact, we were able to reproduce all of the experiments listed in the paper within arange of 1-2 % of accuracy in PyTorch.
Revised: Aside from that fact, we were able to reproduce all of the experiments listed in the paper within arange of 1-2 % of accuracy in PyTorch.

Original: 4.1 Regret Calculation for L2-regularized Logistic Regression
Revised: 4.1 Regret Calculation for L2-regularized Logistic Regression

Original: This particular experiment seeks to establish the ﬁrst claim made by the authors, pertaining to the utilisation ofthe convexity of the problem statement, to yield faster convergence.
Revised: This particular experiment seeks to establish the ﬁrst claim made by the authors, pertaining to the utilisation ofthe convexity of the problem statement, to yield faster convergence.

Original: Across all the datasets in question, SAdamdoes converge fastest, and hence has the lowest overall regret.
Revised: Across all the datasets in question, SAdamdoes converge fastest, and hence has the lowest overall regret.

Original: SC-RMSProp and SC-Adagrad have almost identicalperformances with very minor quantitative differences only in the decimal places.
Revised: SC-RMSProp and SC-Adagrad have almost identicalperformances with very minor quantitative differences only in the decimal places.

Original: It must be noted that the ﬁnal valuesof regret here are higher than the values reported in the actual paper due to the values in the paper being normalizedby the batch size.
Revised: It must be noted that the ﬁnal valuesof regret here are higher than the values reported in the actual paper due to the values in the paper being normalizedby the batch size.

Original: Adam and AMSgrad also have identical performances, however their overall regret is much higher compared to the algorithms that take advantage of the convexity.
Revised: Adam and AMSgrad also have identical performances, however their overall regret is much highercompared to the algorithms that take advantage of the convexity.

Original: OGD due to its simplistic algorithmic setup has thehighest overall regret as expected in all of the datasets.
Revised: OGD due to its simplistic algorithmic setup has thehighest overall regret as expected in all of the datasets.

Original: [Figure]
Revised: [Figure]

Original: 4.2 Performance on 4 Layer CNN
Revised: 4.2 Performance on 4 Layer CNN

Original: [Figure]

Original: The 4 Layer CNN is trained using different optimizers to substantiate the second claim according to which, theperformance of the algorithms is sustained even in case of non convex problems.
Revised: The 4 Layer CNN is trained using different optimizers to substantiate the second claim according to which, theperformance of the algorithms is sustained even in case of non convex problems.

Original: The test accuracy and the traininglosses are reported against all of the datasets.
Revised: The test accuracy and the traininglosses are reported against all of the datasets.

Original: The performance of SAdam is again overall the highest if only testaccuracy is taken into account.
Revised: The performance of SAdam is again overall the highest if only testaccuracy is taken into account.

Original: The performance on CIFAR 10 and MNIST is only marginally better than the next bestoption which is SC-RMSProp.
Revised: The performance on CIFAR 10 and MNIST is only marginally better than the next bestoption which is SC-RMSProp.

Original: However the test accuracy achieved after training for a 100 epochs on the CIFAR 100dataset exceeds the values reported in the actual paper by 2% and is signiﬁcantly higher than all of the other alternatives.
Revised: However the test accuracy achieved after training for a 100 epochs on the CIFAR 100dataset exceeds the values reported in the actual paper by 2% and is signiﬁcantly higher than all of the other alternatives.

Original: In case only training losses are compared, SAdam achieves the lowest values in case of MNIST and CIFAR 100, but for
Revised: In case only training losses are compared, SAdam achieves the lowest values in case of MNIST and CIFAR 100, but for

Original: CIFAR 10, OGD has the lowest loss value after 100 epochs.
Revised: CIFAR 10, OGD has the lowest loss value after 100 epochs.

Original: Amongst the other optimizers, SC-RMSProp consistentlyscores better metrics, with both Adam and AMSGrad again having comparable values following closely behind.
Revised: Amongst the other optimizers, SC-RMSProp consistentlyscores better metrics, with both Adam and AMSGrad again having comparable values following closely behind.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: 4.3 Performance on ResNet- 
Revised: 4.3 Performance on ResNet-

Original: [Figure]
Revised: [Figure]

Original: For this experiment, like the authors, we run the experiment 10 times, and report the mean aggregate values across allruns.
Revised: For this experiment, like the authors, we run the experiment 10 times, and report the mean aggregate values across allruns.

Original: In our results, AmsGrad gives the best results on the test set after 100 epochs, although SAdam does appear togive the best performance after around 30 epochs, before stagnating at the same accuracy levels, since it converges to a0 at that point, and keeps taking random jumps from that point onwards.
Revised: In our results, AmsGrad gives the best results on the test set after 100 epochs, although SAdam does appear togive the best performance after around 30 epochs, before stagnating at the same accuracy levels, since it converges to a0 at that point, and keeps taking random jumps from that point onwards.

Original: OGD suffers from this phenomenon the most,and thus has uneven jumps sporadically throughout its learning phase.
Revised: OGD suffers from this phenomenon the most,and thus has uneven jumps sporadically throughout its learning phase.

Original: However, it must be noted that all the optimizersthat take advantage of the convexity, retain their faster convergence property while training on ResNet as well.
Revised: However, it must be noted that all the optimizersthat take advantage of the convexity, retain their faster convergence property while training on ResNet as well.

Original: [Figure]
Revised: [Figure]

Original: Discussion
Revised: 4.4 Multi Layer LSTM Language Model

Revised: In the language domain, Stochastic Gradient Descent(SGD) when used in conjunction with Momentum gives bestresults on most tasks. For both our regularised and non regularised models, the same observation ensued, with Adamperforming considerably worse than the SGD Momentum combination with perplexities on the test set being higher bya factor of 5 for Adam. The performance of SAdam almost mirrored that of Adam, with marginal differences in theﬁnal perplexities as can be seen in the table below.

Original: [Figure]
Revised: [Table]

Original: SAdam as shown in all of the experiments above, has the property of converging faster specially so in convex functions,but also to a certain extent in non-convex environments.
Revised: SAdam as shown in all of the experiments above, has the property of converging faster specially so in convex functions,but also to a certain extent in non-convex environments.

Original: Even though, the power of the algorithm diminishes with anincrease in the complexity of the non-convex functions, SAdam still performs better that Adam and AmsGrad onlyfalling short behind SC RMSProp in those cases.
Revised: Even though, the power of the algorithm diminishes withan increase in the complexity of the non-convex functions, SAdam still performs better that Adam and AmsGradonly falling short behind SC RMSProp in those cases.

Revised: Certainly its performance in language tasks show that as anoptimizer its utility is not domain agnostic and might suffer heavily in some other non-vision tasks.

Original: Although Adam has been the primary choice in optimizers for a longtime for most deep learning applications, the faster convergence rates of SAdam makes it a viable choice for any convexsetting, and at the very least enables quicker baseline evaluation checks of deeper networks, leading to reduced usage ofhardware acceleration, while choosing amongst many model architectures.
Revised: However since theperformance of SAdam mirrored that of Adam in the same, any domain in which Adam has been shown to work well,working with SAdam should be taken into consideration as well since the faster convergence rates would at the veryleast enable quicker baseline evaluation checks of deeper networks, leading to reduced usage of hardware acceleration,while choosing amongst many model architectures.

Original: 5.1 Strength
Revised: 5.1 Strength

Original: [Figure] Whilst writing our code, we chose to build everything from scratch in a different framework, taking the algorithmsmentioned in the base papers without letting any bias creep in from the source code.
Revised: Whilst writing our code, we chose to build everything from scratch in a different framework, taking the algorithmsmentioned in the base papers without letting any bias creep in from the source code.

Original: We also in addition, conducted theexperiments with various seeds, and the overall results remained the same.
Revised: We also in addition, conducted theexperiments with various seeds, and the overall results remained the same.

Original: We would like to point out that, our codealso serves as an open source contribution for all of the optimizers against which comparisons of SAdam were drawn,none of which except for Adam and AMSGrad had a publicly available PyTorch implementation, at the time of writingthis report. 
Revised: We would like to point out that, our codealso serves as an open source contribution for all of the optimizers against which comparisons of SAdam were drawn,none of which except for Adam and AMSGrad had a publicly available PyTorch implementation, at the time of writingthis report. We do not provide the link here, to respect the integrity of the double blind review process, however ourcode is available for public access on GitHub.

Original: 5.2 Weakness
Revised: 5.2 Weakness

Original: It must be noted that although SAdam performs the best in deeper networks in most of the aforementioned experiments,there is no theoretical foundation for this empirical ﬁnding.
Revised: It must be noted that although SAdam performs the best in deeper networks in most of the aforementioned experiments,there is no theoretical foundation for this empirical ﬁnding.

Original: Any attempt to do so warrants a sophisticated level ofunderstanding of the calculus involved in the optimization mathematics and consequently our ability to address thisconcern was limited.
Revised: Any attempt to do so warrants a sophisticated level ofunderstanding of the calculus involved in the optimization mathematics and consequently our ability to address thisconcern was limited.

Original: 5.3 What was easy
Revised: 5.3 What was easy

Original: The source code provided by the authors worked in one go, without modiﬁcations of any kind required.
Revised: The source code provided by the authors worked in one go, without modiﬁcations of any kind required.

Original: Even, when thebase framework was changed to PyTorch, all the empirical experiments made in the original paper, were relatively easyto setup, right from procuring all the 3 datasets to the selection of the hyperparameters, owing to clear mention of allsuch variables in the paper.
Revised: Even, when thebase framework was changed to PyTorch, all the empirical experiments made in the original paper, were relatively easyto setup, right from procuring all the 3 datasets to the selection of the hyperparameters, owing to clear mention of allsuch variables in the paper.

Original: The algorithms itself, for each of the optimizers were clearly explained, and the translationof the algorithm to code was seamless.
Revised: The algorithms itself, for each of the optimizers were clearly explained, and the translationof the algorithm to code was seamless.

Original: In addition, the low runtime for each individual run meant that, no additionalhardware was required, and training on Google Colab was sufﬁcient for each experiment given in the paper.
Revised: In addition, the low runtime for each individual run meant that, no additionalhardware was required, and training on Google Colab was sufﬁcient for each experiment given in the paper.

Original: 5.4 What was difﬁcult
Revised: 5.4 What was difﬁcult

Original: As mentioned before, implementing the additional optimizers in PyTorch compunded the work at hand, since to provethe efﬁcacy of SAdam itself, a comparison with the state of the art optimizers with the same environmental variables atplay was necessary.
Revised: As mentioned before, implementing the additional optimizers in PyTorch compunded the work at hand, since to provethe efﬁcacy of SAdam itself, a comparison with the state of the art optimizers with the same environmental variables atplay was necessary.

Original: Besides this, the regret calculation experiment for L2-regularized Logistic Regression took muchmore time than expected owing to the contradictions in the proposed values of mini-batch size for each of the datasets,by the authors of SC-RMSProp and SAdam.
Revised: Besides this, the regret calculation experiment for L2-regularized Logistic Regression took muchmore time than expected owing to the contradictions in the proposed values of mini-batch size for each of the datasets,by the authors of SC-RMSProp and SAdam.

Original: Although SAdam works better for a far wider range of possible mini-batchsizes, the performance of SC-RMSProp increases, and eventually even surpasses that of SAdam if the mini-batch sizeis decreased substantially.
Revised: Although SAdam works better for a far wider range of possible mini-batchsizes, the performance of SC-RMSProp increases, and eventually even surpasses that of SAdam if the mini-batch sizeis decreased substantially.

Original: The authors of SC-RMSProp themselves suggest a mini-batch size of 1, which thereforedelivers the best of results for their alogrithms itself.
Revised: The authors of SC-RMSProp themselves suggest a mini-batch size of 1, which thereforedelivers the best of results for their alogrithms itself.

Original: For most real world applications, a per example based approachto learning is more likely to cause unstable optimization, and a sufﬁciently large mini-batch size leads to a smootherconvergence, therefore holistically SAdam should be a preferred choice in a convex setting.
Revised: For most real world applications, a per example based approachto learning is more likely to cause unstable optimization, and a sufﬁciently large mini-batch size leads to a smootherconvergence, therefore holistically SAdam should be a preferred choice in a convex setting.

Original: 5.5 Communication with original authors
Revised: 5.5 Communication with original authors

Original: At the time of the launch of the Reproducibility challenge source code for the paper had not been shared on the openreview portal.
Revised: At the time of the launch of the Reproducibility challenge source code for the paper had not been shared on the openreview portal.

Original: However with our ﬁrst email correspondence, they shared their codebase promptly.
Revised: However with our ﬁrst email correspondence, they shared their codebase promptly.

Original: The discrepancies inthe recommended value of the mini-batch sizes proposed for the different algorithms were discussed with the authors,and they clariﬁed their reasoning behind choosing of a different batch size.
Revised: The discrepancies inthe recommended value of the mini-batch sizes proposed for the different algorithms were discussed with the authors,and they clariﬁed their reasoning behind choosing of a different batch size.

Original: Besides this, the authors answered our minorqueries with regards to the source code as well, and were supportive throughout the entire reproducibility process.
Revised: Besides this, the authors answered our minorqueries with regards to the source code as well, and were supportive throughout the entire reproducibility process.