Paper:
SELF-TUNING NETWORKS: BILEVEL OPTIMIZATION OF HYPERPARAMETERS US- ING STRUCTURED BEST-RESPONSE FUNCTIONS
Hyperparameter optimization can be formulated as a bilevel optimization problem, where the optimal parameters on the training set depend on the hyperparameters. We aim to adapt regularization hyperparameters for neural networks by fitting compact approximations to the best-response function, which maps hyperparameters to optimal weights and biases. We show how to construct scalable best-response approximations for neural networks by modeling the best-response as a single network whose hidden units are gated conditionally on the regularizer. We justify this approximation by showing the exact best-response for a shallow linear network with L 2 -regularized Jacobian can be represented by a similar gating mechanism. We fit this model using a gradient-based hyperparameter optimization algorithm which alternates between approximating the best-response around the current hyperparameters and optimizing the hyperparameters using the approximate best-response function. Unlike other gradient-based approaches, we do not require differentiating the training loss with respect to the hyperparameters, allowing us to tune discrete hyperparameters, data augmentation hyperparameters, and dropout probabilities. Because the hyperparameters are adapted online, our approach discovers hyperparameter schedules that can outperform fixed hyperparameter values. Empirically, our approach outperforms competing hyperparameter optimization methods on large-scale deep learning problems. We call our networks, which update their own hyperparameters online during training, SelfTuning Networks (STNs).

Review:
========\\
Summary\\
========\\
The paper deals with hyper-parameter optimization of neural networks. The authors formulate the problem as a bilevel optimization problem: minimizing the validation loss over the hyperparameters, subject to the parameters being at the minimum of the training loss. The authors propose an approximation of the so-called best-response function, that maps the hyperparameters to the corresponding optimal parameters (w.r.t the minimization of the training loss), allowing a formulate as a single-level optimization problem and the use gradient descent algorithm. The proposed
approximation is based on shifting and scaling the weights and biases of the network. There are no guarantee on its quality except in some very simple cases. The approach assumes a distribution on the hyperparameters, governed by a parameter, which is adapted during the course of the training to achieve a compromise between the flexibility of the best-response function and the quality of its local approximation around the current hyperparameters. The authors show
that their approach beats grid-search, random search and Bayesian optimization on the CIFAR-10 and PTB datasets. They point out that the dynamic update of the hyperparameters during the training allows to reach a better performance than any fixed hyperparameter. \\
======================\\
Comments and questions\\
======================\\
Can cross-validation be adapted to this approach? \\
Can this be used to optimize the learning rate? Which is of course a crucial hyperparameter and that needs an update schedule during the training. \\
Section 3.2:\\
"If the entries are too large, then θ̂ φ will not be flexible enough to capture the best- response over the sampled neighborhood. However, its entries must remain sufficiently large so that θ̂ φ captures the local shape around the current hyperparameter values." Not clear why -- more explanations would be helpful. \\
"minimizing the first term eventually moves all probability mass towards an optimum λ∗ ,resulting in σ = 0". I can't see how minimizing the first term w.r.t \phi (as in section "2.2.Local approximation") would alter \sigma. \\
"τ must be set carefully to ensure...". The authors still do not explain how to set \tau. \\
Section 3.3: \\
If the hyperparameter is discrete and falls in Case 2, then REINFORCE gradient estimator is used. What about the quality of this gradient? \\
Section 5, paragraph Gradient-Based HO: "differentiating gradient descent" needs reformulation -- an algorithm cannot be differentiated. \\
Pros \\
- The paper is pretty clear \\
- Generalizes a previous idea and makes it handle discrete hyperparameters and scale better. \\
- I like the idea of hyperparameters changing dynamically during the training which allows to explore a much larger space than one value \\
- Although limited, the experimental results are convincing \\
Cons \\
- The method itself depends on some parameters and it is not clear how to choose them. Therefore it might be tricky to make it work in practice. I feel like there is a lot of literature around HO but very often people still use the very simple grid/random search, because the alternative methods are often quite complex to implement and make really work. So the fact that the method depends on "crucial" parameters but that are not transparently managed may be a big drawback to its applicability. \\
- No theoretical guarantee on the quality of the used approximation for neural networks \\
- Does not handle the learning rate which is a crucial hyperparameter (but maybe it could) \\