Paper:
As the complexity of neural network models has grown, it has become increasingly important to optimize their design automatically through metalearning. Methods for discovering hyperparameters, topologies, and learning rate schedules have lead to significant increases in performance. This paper shows that loss functions can be optimized with metalearning as well, and result in similar improvements. The method, Genetic Loss-function Optimization (GLO), discovers loss functions de novo, and optimizes them for a target task. Leveraging techniques from genetic programming, GLO builds loss functions hierarchically from a set of operators and leaf nodes. These functions are repeatedly recombined and mutated to find an optimal structure, and then a covariance-matrix adaptation evolutionary strategy (CMA-ES) is used to find optimal coefficients. Networks trained with GLO loss functions are found to outperform the standard cross-entropy loss on standard image classification tasks. Training with these new loss functions requires fewer steps, results in lower test error, and allows for smaller datasets to be used. Loss function optimization thus provides a new dimension of metalearning, and constitutes an important step towards AutoML.

Review:
The authors present a framework to perform meta-learning on the loss used for
training. They introduce the Baikal loss, obtained using the MNIST dataset, and
BaikalCMA where the coefficients have been tuned. The evaluation of these loss
functions is performed on the MNIST and CIFAR-10, and according to the results
they converge faster, towards lower test error and need fewer samples to obtain
results similar to the cross-entropy loss.
The claims are clearly stated and the framework is detailed, the experiments
cover all the potential benefits of the Baikal loss. However it seems that some
potentially critical points have been omitted. The cross-entropy loss is well
known to be beneficial in dataset with severe class imbalance. The two datasets
used for evaluation are perfectly balanced, it might beneficial to see how it
performs in the unbalanced case.
I have a couple of concerns about the method. First about step "(1) loss
function discovery": The initial population starts with trees of depth at most
2, and the final solution(Baikal) has either 2 or 3 (depending on which
definition of depth is chosen). It is unclear that the genetic optimization is
superior to simply choosing random loss functions. I think it would be relevant
to add a figure that shows how the fitness of the leader of each generation
evolves over time.
The second step "(2) coefficient optimization", while objectively generating a
loss function that was superior on the metrics evaluated, raised some
questions. In equation (2) the factor "1.5352" seems to be equivalent to adding
a constant to the loss, which should not impact optimization. Also the factor
"2.7279" seems to be equivalent to a change in learning rate. This may be an
indication that the learning rate search was not done thoroughly. It would be
beneficial to clarify when it is happening: a) For each individual of the
population during step (1), b) before performing CMA, c) after CMA. Also: Was
learning rate search was performed on the network trained with Cross-Entropy? It
was not entirely clear from the experiment details in Appendix A.2.1.
About the Baikal loss itself, I fear that it could produce models that have very
poor calibration, it might be nice to evaluate that (even if it is only in
the appendix).
While the paper does a great job at presenting the problem and its applications
and propose a framework that generated a loss that can transfer to other
datasets without any tuning required. I think it lacks a more thorough
evaluation and description of the dynamics observed during the genetic
evolution, and the performance of the Baikal loss on other datasets (my quick
experients with it on ImageNet diverged I did not have the time necessary to
tune the hyper-parameters).
Minor remarks:
There might be a slight omission in section 3.1: according to Figure 1, exp(x)
is one of the potential unary operators explored by the GLO framework. However
it is not present it the list of operators. Could you clarify this?
To the best of my knowledge, in the machine learning literature, it seems that
the letter x is used to denote the prediction and y for the ground truth. The
fact that this paper used the opposite convention confused me the first time I
read it.