Paper:
Evolutionary Stochastic Gradient Descent for Optimization of Deep Neural Networks
We propose a population-based Evolutionary Stochastic Gradient Descent (ESGD) framework for optimizing deep neural networks. ESGD combines SGD and gradient-free evolutionary algorithms as complementary algorithms in one framework in which the optimization alternates between the SGD step and evolution step to improve the average fitness of the population. With a back-off strategy in the SGD step and an elitist strategy in the evolution step, it guarantees that the best fitness in the population will never degrade. In addition, individuals in the population optimized with various SGD-based optimizers using distinct hyperparameters in the SGD step are considered as competing species in a coevolution setting such that the complementarity of the optimizers is also taken into account. The effectiveness of ESGD is demonstrated across multiple applications including speech recognition, image recognition and language modeling, using networks with a variety of deep architectures.

Review:
The paper combines ES and SGD in a complementary way, not viewing as an alternative to each other. More intuitively, in the course of optimization, the author proposes to use evolutionary strategy to make optimiser adjust at different sections (geometry) of optimisation path. Since the geometry of the loss surface can differ drastically at different locations, it is reasonable to use different gradient based optimisers with different hyper parameters at different locations.  Then the main question becomes how to choose a proper optimiser at different locations. The paper proposes ES for that.   In terms of writing, the paper is readable and easy to follow. Simple comments for better readability are as follow. It would be better if brief explanation about ‘roullette wheel selection’ is given to be more self-contained (in line 138). It seems that psi in lines 145~147 should have superscript (k), the similar ambiguity at the last line of the algorithm1.  ***Related works The paper ‘Population Based Training of Neural Networks (PBT)’ uses the similar method as ESGD. Both uses ES but PBT aims to optimise hyperparameters of neural network while ESGD tries to optimise parameters of neural network. However, it is also possible to view that ESGD optimizes specific hyperparameters (different optimizers at different sections in the course of optimization). Moreover, it looks like that PBT can basically do the similar thing as ESGD with some minor modifications. We can try to optimise optimisers at different sections of the course of optimization as a hyperparameter using PBT.  Since in PBT, the under performing ones are replaced by well performing one together with their parameters, thus evaluation and mutation of parameters in ESGD can be done similarly in PBT. It appears that this work is a special case of PBT.  We hope that the author make the contribution over PBT more clearly. One suggestion is to compare PBT and ESGD to show the benefit of the specific choice of ES in ESGD.  ***Back-off This concept and related theorem is one thing which can differentiate ESGD from PBT. However, back off looks like a commonly used heuristic which is just using the best value in the course of optimization. The statement of the theorem is quite trivial because it basically keeps the improved value. Moreover, it seems that this theorem does not have strong practical implication since the author uses stochastic back off in one experiment, which breaks continuous improvement.  ***Robustness to the choice of ESGD hyperparameters In experiments, the configuration of a couple of ESGD hyperparameters varies for different data set. Pool of optimizers in SGD step - (in CIFAR10 - only using SGD) Back-off - in LM, stochastic back-off with probability 0.7)  Even though a fair amount of engineering hyperparameters is indispensible for a well performing ES, a question is whether ESGD is robust with respect to ESGD hyperparameters or consistent ESGD hyperparameter can be recommended. Some experiment showing the influence of ESGD hyperparameters can be given.  ***Using larger pool of optimizers in SGD step? Since various optimizers like AdaGrad, AdaDelta, RMSProp etc. behave differently, the geometry on which they can optimise well can be different and various optimiser can take care of various geometry. Contrast to the fact that, in CIFAR10, ESGD works better only with SGD, a naive conjecture is that the performance can be improved if various of optimizers are considered to tackle various geometry. If the author has not tried ESGD with various optimizers, it would be interesting to see the result with various optimiser. Or if it has been tried already, then the explanation or guideline to choose pool of optimiser can be beneficial.   ***Including random initialisation to mutation step can improve ESGD? In the proposed algorithm, random initialisation is not used and descendants are selected from parent generation with added perturbation, so there is no chance that strong mutation can happen. Can this strong mutation, which is similar to large perturbation, improve the performance?