S ELF -T UNING N ETWORKS : B ILEVEL O PTIMIZATION OF H YPERPARAMETERS USING S TRUCTURED
B EST -R
ESPONSE
F UNCTIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
Hyperparameter optimization can be formulated as a bilevel optimization problem, where the optimal parameters on the training set depend on the hyperparameters.
We aim to adapt regularization hyperparameters by ﬁtting compact approximations to the best-response function , which maps hyperparameters to optimal weights and biases.
We show that the best-response function for a shallow linear network with L 2 -regularized Jacobian can be represented by a single network whose hidden units are gated conditionally on the regularizer.
Based on this, we propose a gating-based architecture for approximating best-response functions for neural nets more broadly and which can be used as a drop-in replacement for existing deep learning modules.
We ﬁt this model online during training using a gradient-based hyperparameter optimization algorithm which alternates between approximating the best-response around the current hyperparameters and optimizing the hyperparameters using the approximate best-response function.
Unlike other gradient-based approaches, we do not require differentiating the training loss with respect to the hyperparameters, allowing us to tune discrete hyperparameters, data augmentation hyperparameters and dropout probabilities.
Empirically, our approach outperforms competing hyperparameter optimization methods on large-scale deep learning problems.
We call our networks, which update their own hyperparameters online during training, Self-Tuning Networks .
I NTRODUCTION
Regularization hyperparameters such as weight decay, data augmentation, and dropout (Srivastava et al., 2014) are crucial to the generalization of neural networks (NNs), but difﬁcult to tune.
Popular approaches to hyperparameter optimization (HO) include grid search, random search (Bergstra &
Bengio, 2012), and Bayesian optimization (Snoek et al., 2012).
These approaches work well with low-dimensional hyperparameters and ample computational resources.
However, they pose HO as a black-box optimization problem, ignoring structure which can be exploited for faster convergence.
We can formulate HO as a bilevel optimization problem.
Let w and λ denote parameters and hyperparameters, and let L T and L V be functions mapping parameters and hyperparameters to training and validation losses.
We aim to solve 1 :
[Equation]
Substituting the best-response function w ∗ ( λ ) =
arg min w L T
[Equation]
λ , w ) gives a single-level problem:
[Equation]
Gradient-based HO methods attempt to solve Problem 1 by approximating the best-response or
its
Jacobian.
This can be done by differentiating through the training procedure (Maclaurin et al., 2015) or by implicitly deriving the best-response Jacobian (Larsen et al., 1996), but these methods struggle with discrete hyperparameters and are computationally expensive.
Recently, Lorraine & Duvenaud
(2018) proposed directly approximating the best-response by a parametric function ˆ w φ .
They jointly optimized φ and λ , ﬁrst φ so ˆ w φ ≈ w ∗ in a neighborhood around the current hyperparameters, then λ by using ˆ w φ as a proxy for w ∗ in Eq. 2:
[Equation]
This worked for small-scale problems on MNIST using L 2 regularization, but it is difﬁcult to represent and compute ˆ w φ when w is high-dimensional.
We begin by constructing scalable and compact best-response approximations for NNs by analyzing the best-response structure in a Jacobian-regularized two-layer linear network.
Here, the best-response scales the network’s hidden units by an amount depending on the hyperparameters.
Exploiting this structure, we give a memory efﬁcient best-response approximation which scales to deep, nonlinear modern architectures.
Using this approximation gives a gradient-based HO algorithm that is (1) computationally inexpensive, (2) can optimize all regularization hyperparameters — including discrete hyperparameters — and (3) scales to large NNs.
We call the resulting networks Self-Tuning Networks (STNs) since they update their own hyperparameters online during training.
We empirically evaluate the performance of STNs for HO on large-scale deep-learning problems with the PTB and CIFAR-10 datasets.
We compare STN performance to other popular HO approaches and ﬁnd that STNs achieve lower validation loss in less time.
B ILEVEL O PTIMIZATION
A bilevel optimization problem consists of two sub-problems called the upper-level and lower-level problems, where the upper-level problem must be solved subject to optimality of the lower-level problem.
Minimax problems are an example of bilevel programs where the upper-level objective equals the negative lower-level objective.
Bilevel programs were ﬁrst studied in economics to model leader/follower ﬁrm dynamics (Von Stackelberg, 2010) and have since found uses in various ﬁelds (see Colson et al.
for an overview).
In machine learning, many problems can be formulated as bilevel programs; for example, HO, GAN training (Goodfellow et al., 2014), meta-learning, and neural architecture search (Zoph & Le, 2016).
Even if all objectives and constraints are linear, bilevel problems are strongly NP-hard (Hansen et al., 1992; Vicente et al., 1994).
Due to difﬁculty in obtaining exact solutions, most work has focused on restricted settings, considering linear, quadratic, and convex functions.
In contrast, we focus on obtaining local solutions in the nonconvex, differentiable, and unconstrained setting.
Let
F, f : R n × R m → R denote the upper- and lower-level objectives and λ ∈ R n , w ∈ R m denote the upper- and lower-level variables.
We aim to solve:
[Equation]
It is desirable to design a gradient-based algorithm for solving Problem 4, since using gradient information provides drastic speed-ups over black-box optimization methods (Nesterov, 2013).
The simplest method is simultaneous gradient descent, updating λ using ∂F / ∂ λ and w using ∂f / ∂ w .
However, simultaneous gradient descent often gives incorrect solutions since it fails to account for the dependence of w on λ .
Consider the relatively common situation where F doesn’t depend directly on λ , so that ∂F / ∂ λ ≡ 0 and hence λ is never updated.
2. G RADIENT D ESCENT VIA THE B EST -R ESPONSE F UNCTION
A more principled approach to solving Problem 4 is to use the best-response function (Gibbons, 1992).
Assume the lower-level Problem 4b has a unique optimum w ∗ ( λ ) for each λ .
Substituting the best-response function w ∗ converts Problem 4 into a single-level problem:
[Equation] [Equation]
If w ∗ is differentiable and has a unique optimum for Problem 4b for each λ , we can minimize Eq.using gradient descent on F ∗ w.r.t.
λ .
This method requires differentiability of w ∗ and a unique optimum for Problem 4b for each λ .
In general, these conditions are difﬁcult to verify.
We give sufﬁcient conditions for them to hold in a neighborhood of a point ( λ 0 , w 0 ) where w 0 solves Problem 4b given λ 0 .
Lemma 1.
(Fiacco & Ishizuka, 1990) Let w 0 solve Problem equation 4b for λ 0 .
Suppose f is Cin a neighborhood of ( λ 0 , w 0 ) and the Hessian ∂ 2 f / ∂ w 2 ( λ 0 , w 0 ) is positive deﬁnite.
Then for some neighborhood U of λ 0 , there exists a continuously differentiable function w ∗ :
U → R m such that w ∗ ( λ ) is the unique solution to Problem 4b for each λ ∈ U and w ∗ ( λ 0 ) =
w 0 .
Proof.
See Appendix B.1.
The gradient of F ∗ decomposes into two terms, which we term the direct gradient and the response gradient .
The direct gradient captures the direct reliance of the upper-level objective on λ , while the response gradient captures how the lower-level parameter responds to changes in the upper-level parameter:
[Equation]
Even if ∂F / ∂ λ (cid:54)≡ 0 and simultaneous gradient descent is possible, including the response gradient can stabilize optimization by converting the bilevel problem into a single level one, as noted by Metz et al.
(2016) for GAN optimization.
Conversion to a single-level problem ensures the gradient vector ﬁeld is conservative, avoiding pathological issues described by Mescheder et al.
2. A PPROXIMATING THE B EST -R ESPONSE F UNCTION
In general, the solution to Problem 4b is a set, but assuming uniqueness of a solution and differentiability of w ∗ can yield fruitful algorithms in practice.
In fact, gradient-based HO methods can often be interpreted as approximating either the best-response w ∗ or its Jacobian ∂ w ∗ / ∂ λ , as detailed in Section 5.
However, these approaches can be computationally expensive.
When applied to
HO, they often struggle with discrete hyperparameters and stochastic hyperparameters like dropout probabilities, since they require differentiating the training loss w.r.t.
hyperparameters.
Promising approaches to directly approximate w ∗ were proposed by Lorraine & Duvenaud (2018), which are detailed below.
Global Approximation.
The
ﬁrst algorithm proposed by Lorraine & Duvenaud (2018) approximated w ∗ as a differentiable function ˆ w φ with parameters φ .
If w represents NN weights, then the mapping ˆ w φ is a hypernetwork (Schmidhuber, 1992; Ha et al., 2016).
If the distribution p ( λ ) is ﬁxed then gradient descent w
r
t. φ minimizes
[Equation]
If support ( p ) is broad and ˆ w φ is sufﬁciently ﬂexible
, then ˆ w φ
can be used as a proxy for w ∗ in Problem 5, resulting in the following objective
ˆ F :
[Equation]
Local Approximation.
In practice, ˆ w φ is usually insufﬁciently ﬂexible to model w ∗ on support ( p ) .
The second algorithm of Lorraine & Duvenaud (2018) locally approximated w ∗ in a neighborhood around the current upper-level parameter λ .
They set p ( (cid:15) | σ ) to a factorized Gaussian noise distribution with a ﬁxed scale parameter σ ∈ R n + , and φ is found by minimizing
the objective:
[Equation]
Intuitively, the upper-level parameter λ is perturbed by a small amount, so the lower-level parameter learns how to respond.
An alternating gradient descent scheme is used, where φ is updated to minimize equation 9 and λ is updated to minimize equation 8.
This approach worked for problems using L 2 weight decay on MNIST (LeCun et al., 1998).
However, it is unclear if the approach works with different regularizers or scales to larger problems.
It requires ˆ w φ , which is a priori unwieldy for high dimensional w .
It is also unclear how to set σ , which deﬁnes the size of the neighborhood on which φ is trained, or if the approach can be adapted to discrete and stochastic hyperparameters.
S ELF -T UNING N ETWORKS
We ﬁrst construct a best-response approximation ˆ w φ that is memory efﬁcient and scales to large
NNs.
Then, we describe a method to automatically adjust the scale of the neighborhood φ is trained on.
We formally describe our algorithm and discuss how it easily handles discrete and stochastic hyperparameters.
We call the resulting networks, which update their own hyperparameters online during training, self-tuning networks (STNs).
3. E XACT B EST -R ESPONSE FOR T WO -L AYER L INEAR N ETWORKS
When w represents the weights of a large NN, ˆ w φ must be ﬂexible enough to model the bestresponse, while remaining memory-efﬁcient and tractable.
We begin by investigating the structure of the best-response in a simple setting to extract insights which can be applied to more complicated problems.
Consider the regression problem with a 2-layer linear network and weights w =
( Q , s ) ∈ R D × D × R D , which predict targets t from inputs x ∈ R D :
[Equation]
Suppose we use a squared-error loss regularized with an L 2 penalty on the Jacobian ∂y / ∂ x , where the penalty weight λ is mapped with exp to lie in R rather than R + :
[Equation]
Lemma 2.
Let w
=
( Q 0 , s 0 ) , where Q 0 is the change-of-basis matrix to the principal components of the data matrix and s 0 solves the unregularized version of Problem 11 given Q 0 .
Then there exist w , b ∈ R D such that the best-response function 2 w ∗ ( λ ) =
( Q ∗ ( λ ) , s ∗ ( λ )) is:
[Equation]
Here, (cid:12) row indicates row-wise rescaling.
Proof.
See Appendix B.2.
Observe that y ( x ; w ∗ ( λ )) can be viewed as sigmoidal gating of hidden units:
[Equation]
This architecture is shown in Figure 1.
Inspired by this example, we use a similar gating of the hidden units to approximate the best-response for deep, nonlinear networks.
3. L INEAR B EST -R ESPONSE A PPROXIMATIONS
We experimented with several gating mechanisms and found gating by a linear transformation of the hyperparameters worked best.
Our architecture scales and shifts the rows of weights and biases (equivalently, the network’s hidden units) by amounts depending on the hyperparameters.
For each layer’s weight matrix W ∈ R
[Equation]
out
[Equation]
in and bias b ∈ R D out , we approximate the best-response as 3 :
[Equation]
This best-response architecture is tractable to compute and memory-efﬁcient: it requires
D out (2 D in + n ) parameters to represent ˆ W φ and D out (2 + n ) parameters to represent ˆ b φ .
In practice, the approximation can be implemented by simply replacing existing modules in deep learning libraries with “hyper” counterparts which accept an additional vector of hyperparameters 4 .
There is no guarantee this architecture can represent the best-response function in general, but intuitively some hidden units should be (de-)emphasized given stronger regularization.
NN generalization is closely related to the network’s Jacobian norm (Novak et al., 2018), suggesting that the structure of the best-response from Problem 11 may extend to other types of regularization.
[Figure]
This architecture allows the use of different perturbed hyperparameters for each minibatch element since the activations of each batch element can be scaled and shifted separately, enabling ˆ w φ to quickly learn how to respond in a neighborhood around the current hyperparameter values.
To assess how accurately this approximation can model the best-response, we provide theoretical results in the quadratic setting for a slightly more general class of approximations: generic linear transformations.
We show that minimizing equation 9 yields the correct optimal-response gradient, hence generic linear approximations are sufﬁcient in this restricted setting.
Theorem 3.
Suppose f is quadratic with ∂ 2 f / ∂ w 2 (cid:31) 0 and p ( (cid:15) | σ ) is Gaussian with mean 0 and variance σ 2 I .
Suppose ˆ w φ is linear and φ ∗ = arg min
φ ˆ f ( λ , φ , σ ) .
Then we have ˆ w φ / ∂ λ
( λ 0 )
=
∂ w ∗ / ∂
λ
( λ ) .
Proof.
See Appendix B.3.
3. A DAPTING THE H YPERPARAMETER D ISTRIBUTION
The entries of σ , which control the scale of the hyperparameter distribution φ is trained on, are crucial to the STN’s performance.
If the entries are too large, then ˆ w φ will not be ﬂexible enough to capture the best-response over the samples.
However, the entries must remain large enough to force ˆ w φ to capture the local shape around the current hyperparameter values.
We illustrate this in Figure 2.
As the smoothness of the loss landscape changes during training, it may be beneﬁcial to vary σ .
To address these issues we propose adjusting σ during training based on the sensitivity of the upperlevel objective to sampled hyperparameters.
We include an entropy term weighted by τ ∈ [0 , 1] which enlarges entries of σ .
The resulting objective is:
[Equation]
This is similar to a variational inference (VI) objective, where the ﬁrst term is analogous to the negative-log likelihood, but τ (cid:54) = 1 .
As τ ranges from 0 to 1 , our objective interpolates between variational optimization (Staines & Barber, 2012) and VI, as noted by Khan et al.
Similar objectives have been used in the VI literature for better training (Blundell et al., 2015) and representation learning (Higgins et al., 2016).
Minimizing the ﬁrst term on its own eventually moves all probability mass towards an optimum λ ∗ , resulting in σ = 0 .
This compels σ to balance between shifting towards 0 to decrease the ﬁrst term while remaining sufﬁciently high to avoid a heavy entropy penalty.
When benchmarking our algorithm’s performance, we evaluate ˆ F at the deterministic current hyperparameter λ 0 .
This is common practice when using stochastic operations during training, e.g., batch-norm or dropout.
3. T RAINING A LGORITHM
We now describe the complete STN training algorithm and discuss how it can tune hyperparameters which other gradient-based algorithms cannot, such as discrete hyperparameters.
We use an unconstrained parametrization λ ∈ R n of the hyperparameters.
Let r denote the element-wise function which maps λ to the appropriate constrained space, which will involve a non-differentiable discretization for discrete hyperparameters.
Let L T and L V denote training and validation losses which are (possibly stochastic, e.g., if using dropout) functions of the hyperparameters and parameters.
Deﬁne functions f, F by f ( λ , w ) =
[Figure]
L T ( r ( λ ) , w ) and F ( λ , w ) =
L V ( r ( λ ) , w ) .
STNs are trained by a gradient descent scheme which alternates between updating φ for T train steps to minimize ˆ f ( λ , φ , σ ) and updating λ and σ for
T valid steps to minimize ˆ F V ( λ , φ , σ ) .
We give our complete algorithm as Algorithm 1 and show how it can be implemented in code in Appendix G.
The possible non-differentiability of r due to discrete hyperparameters poses no problem.
To differentiate ˆ f w.r.t.
φ , we only require ∂f / ∂ w and ∂ ˆ w φ / ∂ φ .
To differentiate ˆ F V w.r.t.
λ i corresponding to a discrete hyperparameter, there are two cases we must consider:
Case 1: For most regularization schemes, L V does not depend on λ
i directly and hence the only gradient is through ˆ w φ .
Thus, the reparametrization gradient can be used when differentiating ˆ F V .
Case 2: If L V relies explicitly on λ
i , then we can use the REINFORCE gradient estimator (Williams, 1992) to estimate ∂ ˆ F V / ∂λ i .
The number of hidden units is an example of a hyperparameter requiring this approach since it directly affects the validation loss.
E XPERIMENTS
Initialize: Approximation parameters φ , hyperparameters λ while not converged do for t = 1 , . .
, T train do (cid:15) ∼ p ( (cid:15) | σ ) φ ← φ − α 1 ∂∂ φ [ f ( λ + (cid:15) , ˆ w φ ( λ + (cid:15) ))] for t = 1 , . . .
, T valid do (cid:15) ∼ p ( (cid:15) | σ ) ˆ F ( λ , σ ) =
F ( λ + (cid:15) , ˆ w φ ( λ + (cid:15) )) − τ H
[ p ( (cid:15) | σ )]
λ ← λ − α 2 ∂∂ λ (cid:104) ˆ F ( λ , σ ) (cid:105) σ ← σ − α 3 ∂∂ σ (cid:104) ˆ F ( λ , σ ) (cid:105)
We applied our methods to convolutional networks and LSTMs (Hochreiter &
Schmidhuber, 1997), yielding self-tuning
CNNs (ST-CNNs) and self-tuning LSTMS (ST-LSTMs).
To understand the behavior of STNs, we empirically investigated their behavior in a simple setting of tuning a single hyperparameter.
Next, we compared the performance of STNs to commonly used HO methods on the CIFAR10 and PTB datasets.
We did not compare with other gradient-based approaches since they cannot tune discrete hyperparameters or dropout probabilities.
4. H YPERPARAMETER S CHEDULES
When tuning a single hyperparameter,
STNs do not converge to a single ﬁxed hyperparameter.
Instead, STNs discover schedules for adapting the hyperparameters online which outperform using any ﬁxed hyperparameter in training.
We examined this in detail on the PTB corpus (Marcus et al., 1993) using an ST-LSTM to tune the output dropout rate applied to hidden units.
The schedule discovered by an ST-LSTM for output dropout, shown in Figure 3, outperforms the best, ﬁxed output dropout rate (0.68) found by a ﬁne-grained grid search, achieving 83.25 vs 85.
[Table]
validation perplexity.
We argue this is a consequence of the schedule, and not regularizing effects from sampling hyperparameters or the limited capacity of ˆ w φ .
To determine if hyperparameter stochasticity is responsible for improved generalization, we trained a standard LSTM while perturbing its dropout rate around the best value found by grid search.
We perturbed using (1) random Gaussian perturbations, and (2) sinusoid perturbations, which implement a cyclic regularization schedule.
STNs outperform both perturbation methods.
To determine if the limited capacity of ˆ w φ acts as a regularizer, we trained a standard LSTM from scratch using the schedule for output dropout discovered by the ST-LSTM.
Again, it outperforms the best-ﬁxed value, giving evidence that the schedule provides beneﬁt.
The ﬁnal validation and test perplexities of each variant are shown in Table 1.
Next, we show that the schedules discovered by STNs are greedy , i.e., the hyperparameters are adapted to be well-suited for the current training phase.
As shown empirically in Appendix F, lower regularization is best early in training, while higher regularization is better later.
We found the STN schedule implements a curriculum by using a low dropout rate early in training, aiding optimization, and gradually increasing the dropout rate, leading to better generalization.
We show the schedules the STN discovers for different initial dropout rates in Figure 3.
The same schedule is followed no matter what initial rate is used.
[Table]
p = 0 .
68 (Fixed) 85.
p ∼ N (0 . 68 , σ = 0 . 05) 84.
p = 0 .
68 + 0 .
1 sin( kπ ) 85.
p = 0 .
(Converged STN) 88.
STN (Ours) 83.
Following STN Schedule
[Table]
[Figure]
4. L ANGUAGE M ODELLING
We evaluated an ST-LSTM on the PTB corpus (Marcus et al., 1993), which is widely used as a benchmark for RNN regularization due to its small size (Gal & Ghahramani, 2016; Merity et al., 2018; Wen et al., 2018).
We used a 2-layer LSTM with 650 hidden units per layer and 650dimensional word embeddings.
We tuned 7 hyperparameters: variational dropout rates for the input, hidden state, and output; embedding dropout (i.e., setting rows of the embedding matrix to 0 );
DropConnect on the hidden-to-hidden weight matrix; and coefﬁcients α and β that control the strength of activation regularization and temporal activation regularization, respectively.
For LSTM tuning, we obtained the best results when using a ﬁxed perturbation scale of 1 for the hyperparameters.
Additional details about the experimental setup and the role of these hyperparameters can be found in Appendix D.
We compare STNs to grid search, random search, Bayesian optimization
[Figure]
and
Hyperband.
5 Figure 4a shows the best validation perplexity achieved by each method over time.
STNs outperform other methods, achieving lower validation perplexity more quickly.
We show the schedules the STN ﬁnds for each hyperparameter in Figures 4b and 4c.
Hyperband achieves validation perplexity 70.78, while STNs achieve 70.30.
[Figure]
i t
[Figure]
4. I MAGE C LASSIFICATION
[Figure]
We evaluated ST-CNNs on the CIFAR10 (Krizhevsky, 2009) dataset, where it is easy to overﬁt with high-capacity networks.
We used the AlexNet architecture (Krizhevsky et al., 2012), optimizing: (1) discrete architectural hyperparameters of the number of ﬁlters in each convolutional layer; (2) continuous hyperparameters of per-layer activation dropout and input dropout, (3) discrete data augmentation hyperparameters of the length and number of cutout holes (DeVries & Taylor, 2017), and (4) continuous data augmentation hyperparameters controlling the amount of noise to apply to the hue, saturation, brightness, and contrast of an image.
We compared STNs to grid search, random search, and Bayesian optimization.
Figure 5 shows the lowest validation loss achieved by each method over time.
Details of the optimization procedures are in
Appendix E.
The baselines methods decay the learning rate at most 3 times by a factor of 10 each time an epoch is completed without the validation loss improving, after which we halt optimization early.
Again, STNs ﬁnd better hyperparameter conﬁgurations in less time than other methods.
The hyperparameter schedule found by the
STN is shown in Figure 6.
R ELATED W ORK
Bilevel Optimization.
Colson et al.
(2007) is an overview of bilevel problems and Bard (2013) is a comprehensive textbook.
When the objectives/constraints are restricted to be linear, quadratic or convex, a common approach replaces the lower-level problem with its KKT conditions added as constraints for the upper-level problem (Hansen et al., 1992; Vicente et al., 1994).
In the unrestricted setting, our work loosely resembles trust-region methods (Colson et al., 2005), which repeatedly approximate the problem locally using a simpler bilevel program.
In closely related work, Sinha et al.
(2013) used evolutionary techniques to estimate the best-response function iteratively.
Hypernetworks.
First considered by Schmidhuber (1993; 1992), hypernetworks are functions mapping to the weights of an NN.
Predicting weights in CNNs has been developed in various forms
[Figure]
(Denil et al., 2013; Yang et al., 2015).
Ha et al.
(2016) used hypernetworks to generate weights for modern CNNs and RNNs.
Brock et al.
(2017) used hypernetworks to globally approximate a bestresponse for architecture search.
Because the architecture is not optimized during training, they require a large hypernetwork, unlike ours which locally approximates the best-response.
Gradient-Based HO.
There are two main approaches.
The ﬁrst approach approximates w ∗ ( λ 0 ) using w T ( λ 0 , w 0 ) , the value of w after T steps of gradient descent on f w.r.t.
w starting at ( λ 0 , w 0 ) .
The descent steps are differentiated through to approximate ∂ w ∗ / ∂ λ
( λ 0 )
≈ ∂ w T / ∂
λ ( λ 0 , w 0 ) .
This approach was proposed by Domke (2012) and used by Maclaurin et al.
(2015), Luketina et al.
(2016) and Franceschi et al.
The second approach uses the Implicit Function Theorem to derive ∂ w ∗ / ∂ λ ( λ 0 ) under certain conditions.
This was ﬁrst used for HO in NNs (Larsen et al., 1996) and developed further by Pedregosa (2016).
Similar approaches have been used for HO in log-linear models (Foo et al., 2008), kernel selection (Chapelle et al., 2002; Seeger, 2007), and image reconstruction (Kunisch & Pock, 2013; Calatroni et al., 2016).
Both approaches struggle with certain hyperparameters, since they differentiate gradient descent or the training loss w.r.t.
the hyperparameters.
In addition, differentiating gradient descent becomes prohibitively expensive as the number of descent steps increases, while implicitly deriving ∂ w ∗ / ∂ λ requires using Hessianvector products with conjugate gradient solvers to avoid directly computing the Hessian.
Model-Based HO.
A common model-based approach is Bayesian optimization, which models p ( r | λ , D ) , the conditional probability of the performance on some metric r given hyperparameters λ and a dataset D
= { ( λ i , r i ) } .
We can model p ( r | h , D ) with various methods (Hutter et al., 2011; Bergstra et al., 2011; Snoek et al., 2012; 2015).
D is constructed iteratively, where the next λ to train on is chosen by maximizing an acquisition function C ( λ ; p ( r | λ , D )) which balances exploration and exploitation.
Training each model to completion can be avoided if assumptions are made on learning curve behavior (Swersky et al., 2014; Klein et al., 2016).
These approaches require building inductive biases into p ( r | λ , D ) which may not hold in practice, do not take advantage of the NN structure when used for HO, and do not scale well with the number of hyperparameters.
However, these approaches have consistency guarantees in the limit, unlike ours.
Model-Free HO.
Model-free approaches include grid-search and random search.
Bergstra & Bengio (2012) advocated using random search over grid search.
Successive Halving (Jamieson & Talwalkar, 2016) and Hyperband (Li et al., 2016) extend random search by adaptively allocating resources to promising conﬁgurations using multi-armed bandit techniques.
These methods ignore structure in the problem, unlike ours which uses of rich gradient information.
However, it is trivial to parallelize model-free methods over computing resources.
Hyperparameter Scheduling.
Population Based Training (PBT) (Jaderberg et al., 2017) considers schedules for hyperparameters.
In PBT, a population of NNs are trained in parallel.
The hyperparameters and weights of under-performing NNs are periodically replaced by better-performing counterparts, which are randomly perturbed before copying.
Thus, a single model can experience different hyperparameters over the course of training, implementing a schedule.
STNs replace the population of NNs by a single best-response approximation and use gradients to tune hyperparameters during a single training run.
C ONCLUSION
We proposed Self-Tuning Networks (STNs), which efﬁciently approximate the best-response of parameters to hyperparameters by scaling and shifting their hidden units.
This allowed us to use gradient-based optimization to tune various regularization hyperparameters, including discrete hyperparameters.
We validated the approach on large-scale problems by showing STNs achieve better generalization performance than competing approaches in less time.
We believe STNs offer a compelling path towards large-scale, automated hyperparameter tuning for neural networks
A T ABLE OF N OTATION
[Table]
A function which reparametrizes hyperparameters to lie in the reals
[Table]
L T ( r ( λ
[Table]
w ))
[Table]
L V ( r ( λ , w )))
=
( f ( λ , w ) , F ( λ
[Table]
w ))
[Table]
predicted
[Table]
elementary
[Table]
multiplication
[Table]
A basis change matrix to the principle components of the data matrix and solution to Problem
[Table]
The basis change matrix and solution to the unregularized Problem
[Table]
The basis matrix and solution for a hyperparameter Problem
[Table]
The data inputs which have had their bases changed by Q
[Table]
The
[Table]
The
[Table]
The (
[Table]
loss) response
[Table]
The hyperparameter gradient
[Table]
a sum of the validation losses direct and response gradients
[Table]
B P ROOFS B. L EMMA
Because w 0 solves Problem 4b given λ 0 , by the ﬁrst-order optimality condition, we must have:
[Equation]
The Jacobian of ∂f / ∂ w decomposes as a block matrix with subblocks given by:
[Equation]
We know f is C 2 in some neighborhood of ( λ 0 , w 0 ) , so ∂f / ∂ w is continuously differentiable in this neighborhood.
By assumption, the Hessian ∂ 2 f / ∂ w 2 is positive deﬁnite and hence invertible at ( λ 0 , w 0 ) .
By the Implicit Function Theorem, there exists a neighborhood V of λ 0 and a unique continuously differentiable function w ∗ :
V → R m such that ∂f / ∂ w ( λ , w ∗ ( λ ))
=
for λ ∈ V and w ∗ ( λ 0 )
=
w 0 .
Furthermore, by continuity we know there is a neighborhood W 1 × W 2 of ( λ 0 , w 0 ) such that ∂ 2 f / ∂ wis positive deﬁnite on this neighborhood.
Setting U
=
V ∩ W 1 ∩ ( w ∗ ) − 1 ( W 2 ) , we can conclude that ∂
2 f / ∂ w 2 ( λ , w ∗ ( λ ))
(cid:31) 0 for all λ ∈ U .
Combining this with ∂f / ∂ w ( λ , w ∗ ( λ ))
=
and using secondorder sufﬁcient optimality conditions, we conclude that w ∗ ( λ ) is the unique solution to Problem 4b for all λ ∈ U .
B. L EMMA
This discussion mostly follows from Hastie et al.
We let X ∈ R N × D denote the data matrix where N is the number of training examples, and D is the dimensionality of the data.
We let t ∈ R N denote the associated targets.
We can write the SVD decomposition of X as:
[Equation]
where U and V are N × D and D × D orthogonal matrices and D is a diagonal matrix with entries d 1 ≥ d 2 ≥ · · · ≥ d D ≥ 0 .
We next simplify the function y ( x ; w ) by setting v = s T Q , so that y ( x ; w ) =
s T
Qx = v T x .
We see that the Jacobian ∂y / ∂ x ≡ v is constant, and Problemsimpliﬁes to standard L 2 -regularized least-squares linear regression with the following loss function:
[Equation]
It is well-known (see Hastie et al.
Chapter 3) that the optimal solution v ∗ ( λ ) minimizing
Equation 18 is given by:
[Equation]
Furthermore, the optimal solution v ∗ to the unregularized version of Problem 18 is given by:
[Equation]
Recall that we deﬁned Q
= V T , i.e. the change-of-basis matrix from the standard basis to the principal components of the data matrix, and we deﬁned s 0 to solve the unregularized regression problem given Q 0 .
Thus, we require that Q T 0 s
= v ∗ which implies s
=
D − 1 U T t .
There are not unique solutions to Problem 11, so we take any functions Q ( λ ) , s ( λ ) which satisfy Q ( λ ) T s ( λ )
= v ∗ ( λ ) as “best-response functions”.
We will show that our chosen functions
Q ∗ ( λ ) = σ ( λ w + b ) (cid:12) Q 0 and s ∗ ( λ ) =
s 0 , where w = 0 and b i = 2 log( d i ) for i = 1 , . .
, D , meet this criteria.
We start by noticing that for any d ∈ R
+ , we have:
[Equation]
It follows that:
[Equation]
B. T HEOREM
By assumption, f is quadratic so there exist A ∈ R n × n , B ∈ R n × m , C ∈ R m × m and d ∈ R n , e ∈ R m such that:
[Equation]
Furthermore, since we assume ∂ 2 f / ∂ w 2 (cid:31) 0 , we must have C (cid:31) 0 .
One can easily compute that:
[Equation]
Setting the derivative equal to 0 and using second-order sufﬁcient conditions, we have:
[Equation]
Hence, we ﬁnd:
[Equation]
We can substitute ˆ w φ ( λ ) =
Uλ + b and the deﬁnition of f into equation 9 to obtain:
[Equation]
Expanding above
[Equation]
we can ﬁnd that equation 34 is equal to:
[Equation] [Equation]
where we have:
[Equation]
We can simplify these expressions considerably by using linearity of expectation and that (cid:15) ∼ p (
[Equation]
| σ ) has mean 0 .
Doing so, then differentiating and making use of various matrix-derivative equalities (Duchi, 2007), we ﬁnd:
[Equation]
Setting the derivative ∂e / ∂ b ( U , b , σ ) equal to 0 , we have:
[Equation]
[Equation]
Setting the derivative for ∂e / ∂
U ( U , b , σ ) equal to 0 , we have:
[Equation]
Substituting the expression for b given by equation 42 into equation 43 and simplifying gives:
[Equation]
This is exactly the best-response Jacobian ∂ w ∗ / ∂ λ
( λ ) as given by equation 33.
Substituting U =
C − 1 B into the equation 42 gives:
[Equation]
This is w ∗ ( λ ) − ∂ w ∗ / ∂ λ ( λ ) , as claimed.
B. B EST -R ESPONSE G RADIENT L EMMA
Lemma 4.
Under the same conditions as Lemma 1 and using the same notation, for all λ ∈ U , we have that:
[Equation]
Proof.
Deﬁne ι ∗ : U → R n × R m by ι ∗ ( λ ) =
( λ , w ∗ ( λ )) .
By ﬁrst-order optimality conditions, we know:
[Equation]
Hence, for all λ ∈ U :
[Equation]
Rearranging gives Equation 48.
C B EST -R ESPONSE A PPROXIMATIONS FOR C ONVOLUTIONAL F ILTERS
We let L denote the number of layers, C l the number of channels in layer l ’s feature map, and K l the size of the kernel in layer l .
We let W l,c ∈ R C l − 1 × K l × K l and b l,c ∈ R denote the weight and bias respectively of the c th convolution kernel in layer l (so c ∈ { 1 , . . .
, C l } ).
For u l,c , a l,c ∈ R n , we deﬁne best-response approximations
ˆ W l,c and ˆ b l,c by:
[Equation] [Equation]
Thus, the best-response parameters used for modeling
[Equation]
are φ l,
c
= { u l,c , a l,c , W l,c scale , W l,c shift , b l,c scale , b l,c shift } .
We can compute the number of parameters used as | φ l,c | = 2 n + 2( | W l,c | + | b l,c | ) .
Summing over channels c , we ﬁnd the total number of parameters is 2 nC l + 2 p , where p is the total number of parameters in the normal CNN layer.
Hence, we use twice the number of parameters in a normal CNN, plus an overhead depending on the number of hyperparameters.
For an implementation in code, see Appendix G.
D L ANGUAGE M ODELLING E XPERIMENT D ETAILS
Here we present additional details on the setup of our LSTM experiments on PTB, and on the role of each hyperparameter we tune.
We tune variational dropout (re-using the same dropout mask for each step in a sequence) on the input to the LSTM, the hidden state between the LSTM layers, and the output of the LSTM.
We also tune embedding dropout, which sets entire rows of the word embedding matrix to 0, effectively removing certain words from all sequences.
We regularize the hidden-to-hidden weight matrix using
DropConnect (zeroing out weights rather than activations).
Finally, we use activation regularization (AR) and temporal activation regularization (TAR).
AR penalizes large activations, and is deﬁned as:
[Equation]
55) where m is a dropout mask and h t is the output of the LSTM at time t .
TAR is a slowness regularizer, deﬁned as:
[Equation]
We tune the scaling coefﬁcients α and β .
E I MAGE C LASSIFICATION E XPERIMENT D ETAILS
Here we present additional details on the CNN experiments.
For all results we hold out 20% of the data for validation.
Our optimization algorithm for the elementary model in the baselines is as follows: We use SGD with a learning rate of 0.01, momentum of 0.9, mini-batches of size 128, and run for at mostepochs.
Our optimization algorithm for the ST-CNN is as follows:
The ST-CNN is optimized with the same algorithm as the elementary model in the baselines.
In other words, we use SGD with learning rate 0.01, momentum of 0.9, mini-batches of size 128.
We use the same optimization algorithm as the ST-LSTM for the hyperparameters.
This means the hyperparameters are optimized with Adam using a learning rate of 0.003.
We alternate between training the best-response approximation and hyperparameters with the same schedule as the ST-LSTM - i.e. T train = 2 steps on the training step and T valid = 1 steps on the validation set.
Additionally, we use an entropy weight of 0 during training when optimizing the scale of the hyperparameter distributions.
F A DDITIONAL D ETAILS ON H YPERPARAMETER S CHEDULES
Here, we draw connections between the hyperparameter schedules learned by STNs and curriculum learning.
Curriculum learning is an instance of a family of continuation methods (Allgower &
Georg, 2012), which optimize non-convex functions by solving a sequence of functions that are ordered by increasing difﬁculty.
In a continuation method, one considers a family of training criteria
C λ ( w ) with parameter λ , where C 1 ( w ) is the true/ﬁnal objective we wish to minimize, and C 0 ( w ) represents the training criterion for a simpler version of the problem.
One starts by optimizing
C 0 ( w ) and then gradually increases λ from 0 to 1, while keeping w at a local minimum of C λ ( w ) .
When training an RL-NTM (Zaremba & Sutskever, 2015), the authors maintain a distribution over the complexity c of a problem instance (deﬁned as the maximum length of the target output for each input), which is shifted towards harder problems as the performance of the model improves.
Figure 7 shows the validation perplexity for different sets of input and output dropout, at various points during training.
The x- and y-axes represent the values of input and output dropout, respectively.
We see that at the start of training, the best validation loss is achieved with small values of both input and output dropout.
As we train for more epochs, the best validation performance is achieved with larger dropout rates.
[Figure]
Figure 8 shows the perturbed values for output dropout we use to investigate if the improved performance yielded by STNs is due to the regularization effect, and not the schedule
[Figure]
[Figure]
We can observe the beneﬁts of greedy hyperparameter schedules by forming a schedule based on the performance in time of models from a grid search over output dropout.
As shown in Figure 9, the schedule formed by taking the best output dropout value in each epoch yields better generalization than any of the ﬁxed hyperparameter values from the initial grid search
[Figure] [Figure]
G C ODE L ISTINGS
In this section we provide PyTorch code listings for the approximate best-response layers used to construct ST-LSTMs and ST-CNNs: the HyperLinear and HyperConv2D classes.
We also provide a stylized version of the optimization steps used on the training set and validation set
[Table]
return output
Listing 2: HyperConv2d used as a drop-in replacement for Conv2d modules class HyperConv2d(nn.
Module):
def __init__(self, in_channels, out_channels, kernel_size, padding, num_hparams, stride=1,bias=True):super(HyperConv2d,self).__init__() self.in_channels = in_channels self.out_channels = out_channels self.kernel_size =
kernel_size self.padding =
padding self.num_
[Table]
= num_
[Table]
self.
[Table]
= stride self.
[Table] [Table]
nn.
[Table]
(torch.
[Table]
( out_channels, in_channels, kernel
[Table]
size, kernel_size)) self.hnet_
[Table]
nn.
[Table]
(torch.
[Table]
( out_channels, in_channels, kernel_size, kernel_size))
if bias: self.elem
[Table]
bias = nn.Parameter(torch.
Tensor(out
[Table]
channels)) self.hnet
[Table]
bias = nn.Parameter(torch.
Tensor(out
[Table]
channels)) else:self.register_parameter(’elem_bias’, None) self.register_parameter(
[Table]
bias’, None) self.htensor_to_scalars = nn.
Linear
[Table]
.num
[Table] [Table]
.out
[Table]
channels*2, bias=False) self.elem
[Table]
= nn.Parameter(torch.ones(1))
self.init_params() def forward(self, input, htensor): """
Arguments:input(tensor): size should be (B, C, H, W) htensor (tensor): size should be (B, D) """ output = F.conv2d(input, self.elem_weight, self.elem_bias,
[Table]
=self.padding,
[Table]
=self.
[Table]
) output *= self.elem_scalar if htensor is not None:
[Table]
to
[Table] [Table]
[Table] [Table]
if self.hnet_bias is not None: hnet_
[Table]
+= (hnet_bscalars * self.
[Table] [Table]
[Table] [Table]
(2) output += hnet_out return output def init_params(self):n=
self.in_channels *
[Table] [Table]
.kernel_
[Table]
stdv
[Table]
. / math
[Table]
(n) self.
[Table]
# Run everything through the model and do gradient descent.
pred = hyper_cnn(images, batch_htensor, hparam_tensor) xentropy_loss
= F.cross_entropy(pred, labels) xentropy_loss.backward() cnn_optimizer.step()
Listing validation scale # Perturb hyperparameters around current value in unconstrained # parametrization, so we can assess sensitivity of validation # loss to the scale of the noise.
batch_htensor =
perturb(htensor, hscale) # Apply necessary reparametrization of hyperparameters.
hparam_tensor = hparam_transform(batch_htensor)
# Get next batch of examples and run through the model.
images, labels =
next_batch(valid_dataset) pred = hyper_cnn(images, batch_htensor, hparam_tensor) xentropy_loss
= F.cross_entropy(pred, labels)
# Add extra entropy weight term to loss.
entropy = compute_entropy(hscale) loss =
xentropy_loss - args.entropy_weight * entropy loss.backward() # Tune the hyperparameters.
hyper_optimizer.step() # Tune the scale of the noise applied to hyperparameters.
scale_optimizer.step()
H A BLATION S TUDIES
In this section we present experiments that show how sensitive our STN models are to different meta-parameters.
In particular, we investigate the effect of using alternative schedules (Figure 10) for the number of optimization steps performed on the training and validation sets.
[Figure]
Additionally, we investigate the effect of using different initial perturbation scales for the hyperparameters, which are either ﬁxed or tuned (Figure 11)
[Figure] [Figure]
Finally
[Figure]
we have included experiments which allow us to assess the importance of the response gradient 12.
This is done by either including or not including the response gradient.
We omit the response gradient by not propagating the hyperparameter gradient through the hypernetwork.
[Figure]