Original: S TOCHASTIC H YPERPARAMETER O PTIMIZATION THROUGH H YPERNETWORKS
Revised: S TOCHASTIC H YPERPARAMETER O PTIMIZATION THROUGH H YPERNETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Machine learning models are usually tuned by nesting optimization of model weights inside the optimization of hyperparameters.
Revised: Machine learning models are usually tuned by nesting optimization of model weights inside the optimization of hyperparameters.

Original: We give a method to collapse this nested optimization into joint stochastic optimization of both weights and hyperparameters.
Revised: We give a method to collapse this nested optimization into joint stochastic optimization of both weights and hyperparameters.

Original: Our method trains a neural network to output approximately optimal weights as a function of hyperparameters.
Revised: Our method trains a neural network to output approximately optimal weights as a function of hyperparameters.

Original: We show that our method converges to locally optimal weights and hyperparameters for sufﬁciently large hypernets.
Revised: We show that our method converges to locally optimal weights and hyperparameters for sufﬁciently large hypernets.

Original: We compare this method to standard hyperparameter optimization strategies and demonstrate its effectiveness for tuning thousands of hyperparameters.
Revised: We compare this method to standard hyperparameter optimization strategies and demonstrate its effectiveness for tuning thousands of hyperparameters.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Hyperparameter λ
Revised: Hyperparameter λ

Original: L o ss
Revised: L o ss

Original: L
Revised: L

Original: Train loss of optimized weights
Revised: Train loss of optimized weights

Original: Train loss of hypernet weights
Revised: Train loss of hypernet weights

Original: Valid.
Revised: Valid.

Original: loss of optimized weights
Revised: loss of optimized weights

Original: Valid.
Revised: Valid.

Original: loss of hypernet weights
Revised: loss of hypernet weights

Original: Optimal hyperparameter λ
Revised: Optimal hyperparameter λ

Original: Model selection and hyperparameter tuning is a major bottleneck in designing predictive models.
Revised: Model selection and hyperparameter tuning is a major bottleneck in designing predictive models.

Original: Hyperparameter optimization can be seen as a nested optimization: The inner optimization ﬁnds model parameters w which minimizes the training loss L Train given hyperparameters λ .
Revised: Hyperparameter optimization can be seen as a nested optimization: The inner optimization ﬁnds model parameters w which minimizes the training loss L Train given hyperparameters λ .

Original: The outer optimization chooses λ to minimize a validation loss L Valid . :
Revised: The outer optimization chooses λ to minimize a validation loss L Valid . :

Original: [Equation]
Revised: [Equation]

Original: Standard practice in machine learning solves (1) by gradient-free optimization of hyperparameters, such as grid search, random search, or Bayesian optimization.
Revised: Standard practice in machine learning solves (1) by gradient-free optimization of hyperparameters, such as grid search, random search, or Bayesian optimization.

Original: Each set of hyperparameters is evaluated by reinitializing weights and training the model to completion.
Revised: Each set of hyperparameters is evaluated by reinitializing weights and training the model to completion.

Original: This is wasteful, since it trains the model from scratch each time, even if the hyperparameters change a small amount.
Revised: This is wasteful, since it trains the model from scratch each time, even if the hyperparameters change a small amount.

Original: Hyperband (Li et al., 2016) and freezethaw Bayesian optimization (Swersky et al., 2014) resume model training and do not waste this effort.
Revised: Hyperband (Li et al., 2016) and freezethaw Bayesian optimization (Swersky et al., 2014) resume model training and do not waste this effort.

Original: Furthermore, gradient-free optimization scales poorly beyond 10 or 20 dimensions.
Revised: Furthermore, gradient-free optimization scales poorly beyond 10 or 20 dimensions.

Original: How can we avoid re-training from scratch each time?
Revised: How can we avoid re-training from scratch each time?

Original: We usually estimate the parameters with stochastic optimization, but the true optimal parameters are a deterministic function of the hyperparameters λ :
Revised: We usually estimate the parameters with stochastic optimization, but the true optimal parameters are a deterministic function of the hyperparameters λ :

Original: [Equation]
Revised: [Equation]

Original: We propose to learn this function .
Revised: We propose to learn this function .

Original: Speciﬁcally, we train a neural network with inputs of hyperparameters, and outputs of an approximately optimal set of weights given the hyperparameters.
Revised: Speciﬁcally, we train a neural network with inputs of hyperparameters, and outputs of an approximately optimal set of weights given the hyperparameters.

Original: This provides two major beneﬁts: First, we can train the hypernet to convergence using stochastic gradient descent, denoted SGD, without training any particular model to completion.
Revised: This provides two major beneﬁts: First, we can train the hypernet to convergence using stochastic gradient descent, denoted SGD, without training any particular model to completion.

Original: Second, differentiating through the hypernet allows us to optimize hyperparameters with gradient-based stochastic optimization.
Revised: Second, differentiating through the hypernet allows us to optimize hyperparameters with gradient-based stochastic optimization.

Original: P a r a m e t e r w
Revised: P a r a m e t e r w

Original: H yp e r p a r a m e t e r λ
Revised: H yp e r p a r a m e t e r λ

Original: L o ss
Revised: L o ss

Original: L T r a i n ( w , λ )
Revised: L T r a i n ( w , λ )

Original: L (w ∗ ( λ ) ,λ )
Revised: L (w ∗ ( λ ) ,λ )

Original: L (w φ ∗ ( λ ) ,λ ) ˆ λ w ∗ (ˆ λ ) , w φ ∗ (ˆ λ )
Revised: L (w φ ∗ ( λ ) ,λ ) ˆ λ w ∗ (ˆ λ ) , w φ ∗ (ˆ λ )

Original: P a r a m e t e r w
Revised: P a r a m e t e r w

Original: H yp e r p a r a m e t e r λ
Revised: H yp e r p a r a m e t e r λ

Original: L o ss
Revised: L o ss

Original: L V a li d . ( w ) λ ∗ w ∗ ( λ ∗ )
Revised: L V a li d . ( w ) λ ∗ w ∗ ( λ ∗ )

Original: λ φ ∗ w φ ∗ ( λ φ ∗ )
Revised: λ φ ∗ w φ ∗ ( λ φ ∗ )

Original:  T RAINING A NETWORK TO OUTPUT OPTIMAL WEIGHTS
Revised:  T RAINING A NETWORK TO OUTPUT OPTIMAL WEIGHTS

Original: How can we train a neural network to output approximately optimal weights of another neural network?
Revised: How can we train a neural network to output approximately optimal weights of another neural network?

Original: A neural net which outputs the weights of another neural net is called a hypernet (Ha et al., 2016).
Revised: A neural net which outputs the weights of another neural net is called a hypernet (Ha et al., 2016).

Original: The basic idea is that at each iteration, we ask a hypernet to output a set of weights given some hyperparameters: w = w φ ( λ ) .
Revised: The basic idea is that at each iteration, we ask a hypernet to output a set of weights given some hyperparameters: w = w φ ( λ ) .

Original: Instead of updating weights w using the loss gradient ∂ L (w) / ∂ w , we update the hypernet weights φ using the chain rule: ∂ L (w φ ) ∂ w φ ∂ w φ ∂φ .
Revised: Instead of updating weights w using the loss gradient ∂ L (w) / ∂ w , we update the hypernet weights φ using the chain rule: ∂ L (w φ ) ∂ w φ ∂ w φ ∂φ .

Original: We call this method hyper-training and contrast it with standard training methods in Figure 3.
Revised: We call this method hyper-training and contrast it with standard training methods in Figure 3.

Original: We call the function w ∗ ( λ ) that outputs optimal weights for hyperparameters a best-response function (Fudenberg & Levine, 1998).
Revised: We call the function w ∗ ( λ ) that outputs optimal weights for hyperparameters a best-response function (Fudenberg & Levine, 1998).

Original: At convergence, we want our hypernet w φ ( λ ) to closely match the best-response function.
Revised: At convergence, we want our hypernet w φ ( λ ) to closely match the best-response function.

Original: 2. A DVANTAGES OF HYPERNET - BASED OPTIMIZATION
Revised: 2. A DVANTAGES OF HYPERNET - BASED OPTIMIZATION

Original: We can compare hyper-training to other model-based hyperparameter schemes, such as Bayesian optimization.
Revised: We can compare hyper-training to other model-based hyperparameter schemes, such as Bayesian optimization.

Original: Bayesian optimization (Snoek et al., 2012) builds a model of the validation loss as a function of hyperparameters, usually using a Gaussian process (Rasmussen & Williams, 2006) to track uncertainty.
Revised: Bayesian optimization (Snoek et al., 2012) builds a model of the validation loss as a function of hyperparameters, usually using a Gaussian process (Rasmussen & Williams, 2006) to track uncertainty.

Original: This approach has several disadvantages compared to hyper-training.
Revised: This approach has several disadvantages compared to hyper-training.

Original: First, obtaining data for standard Bayesian optimization requires optimizing models from initialization for each set of hyperparameters.
Revised: First, obtaining data for standard Bayesian optimization requires optimizing models from initialization for each set of hyperparameters.

Original: In contrast, hyper-training never needs to fully optimize any one model.
Revised: In contrast, hyper-training never needs to fully optimize any one model.

Original: Second, standard Bayesian optimization treats the validation loss as a black-box function: ˆ
Revised: Second, standard Bayesian optimization treats the validation loss as a black-box function: ˆ

Original: L Valid .
Revised: L Valid .

Original: ( λ ) =
Revised: ( λ ) =

Original: f ( λ ) .
Revised: f ( λ ) .

Original: In contrast, hyper-training takes advantage of the fact that the validation loss is a known, differentiable function: ˆ L Valid .
Revised: In contrast, hyper-training takes advantage of the fact that the validation loss is a known, differentiable function: ˆ L Valid .

Original: ( λ ) =
Revised: ( λ ) =

Original: L Valid .
Revised: L Valid .

Original: (w φ ( λ ))
Revised: (w φ ( λ ))

Original: This removes the need to learn a model of the validation loss.
Revised: This removes the need to learn a model of the validation loss.

Original: This function can also be evaluated stochastically by sampling points from the validation set.
Revised: This function can also be evaluated stochastically by sampling points from the validation set.

Original: What sort of parameters can be optimized by our approach?
Revised: What sort of parameters can be optimized by our approach?

Original: Hyperparameters typically fall into two broad categories: 1) Optimization hyperparameters such as learning rates and initialization schemes, and 2) Regularization or model architecture parameters.
Revised: Hyperparameters typically fall into two broad categories: 1) Optimization hyperparameters such as learning rates and initialization schemes, and 2) Regularization or model architecture parameters.

Original: Hyper-training does not have inner optimization hyperparameters because there is no inner training loop, so we can not optimize these.
Revised: Hyper-training does not have inner optimization hyperparameters because there is no inner training loop, so we can not optimize these.

Original: We must still choose optimization parameters for the fused optimization loop, but this is also the case for any model-based hyperparameter optimization method.
Revised: We must still choose optimization parameters for the fused optimization loop, but this is also the case for any model-based hyperparameter optimization method.

Original: The method can be applied to unconstrained bi-level optimization problems which consist of an inner loss function, inner parameter, outer loss function, and an outer parameter.
Revised: The method can be applied to unconstrained bi-level optimization problems which consist of an inner loss function, inner parameter, outer loss function, and an outer parameter.

Original: 1: for i = 1 , . . .
Revised: 1: for i = 1 , . . .

Original: , T outer 2: initialize w 3: λ = hyperopt (cid:0) . . .
Revised: , T outer 2: initialize w 3: λ = hyperopt (cid:0) . . .

Original: , λ ( i ) , L Valid .
Revised: , λ ( i ) , L Valid .

Original: (cid:0) w ( i ) (cid:1)(cid:1) 4: for T inner steps 5: x ∼ Training data 6: w = w − α ∇ w L Train ( x , w , λ ) 7: λ i , w i = λ, w 8: for i = 1 , . . .
Revised: (cid:0) w ( i ) (cid:1)(cid:1) 4: for T inner steps 5: x ∼ Training data 6: w = w − α ∇ w L Train ( x , w , λ ) 7: λ i , w i = λ, w 8: for i = 1 , . . .

Original: T outer 9: if L Valid .
Revised: T outer 9: if L Valid .

Original: (cid:0) w ( i ) (cid:1) <
Revised: (cid:0) w ( i ) (cid:1) <

Original: L Valid .
Revised: L Valid .

Original: (w) then 10: ˆ λ, w = λ
Revised: (w) then 10: ˆ λ, w = λ

Original: i , w i 11: return ˆ λ, w pernet, then hyperparameters 1: 2: initialize φ 3: initialize ˆ λ 4: for T hypernet steps 5: x ∼ Training data, λ ∼ p ( λ ) 6: φ = φ − α ∇ φ L Train ( x , w φ ( λ ) , λ ) 7: 8: for T hyperparameter steps 9: x ∼ Validation data 10: ˆ λ =
Revised: i , w i 11: return ˆ λ, w pernet, then hyperparameters 1: 2: initialize φ 3: initialize ˆ λ 4: for T hypernet steps 5: x ∼ Training data, λ ∼ p ( λ ) 6: φ = φ − α ∇ φ L Train ( x , w φ ( λ ) , λ ) 7: 8: for T hyperparameter steps 9: x ∼ Validation data 10: ˆ λ =

Original: ˆ λ − β ∇ ˆ λ L Valid .
Revised: ˆ λ − β ∇ ˆ λ L Valid .

Original: ( x , w φ (ˆ λ )) 11: return ˆ λ, w φ (ˆ λ )
Revised: ( x , w φ (ˆ λ )) 11: return ˆ λ, w φ (ˆ λ )

Original: 2. L IMITATIONS OF HYPERNET - BASED OPTIMIZATION
Revised: 2. L IMITATIONS OF HYPERNET - BASED OPTIMIZATION

Original: Hyper-training can handle discrete hyperparameters but does not offer special advantages for optimizing over continuous hyperparameters.
Revised: Hyper-training can handle discrete hyperparameters but does not offer special advantages for optimizing over continuous hyperparameters.

Original: The hyperparameters being optimized must affect the training loss - this excludes optimization hyperparameters like learning rate.
Revised: The hyperparameters being optimized must affect the training loss - this excludes optimization hyperparameters like learning rate.

Original: Also, our approach only proposes making local changes to the hyperparameters and does not do uncertainty-based exploration.
Revised: Also, our approach only proposes making local changes to the hyperparameters and does not do uncertainty-based exploration.

Original: Uncertainty can be incorporated into the hypernet by using stochastic variational inference as in Blundell et al.
Revised: Uncertainty can be incorporated into the hypernet by using stochastic variational inference as in Blundell et al.

Original: , but we leave this for future work.
Revised: , but we leave this for future work.

Original: Finally, it is not obvious how to choose the distribution over hyperparameters p ( λ ) .
Revised: Finally, it is not obvious how to choose the distribution over hyperparameters p ( λ ) .

Original: If we do not sample a sufﬁcient number of hyperparameters we may reach sub-optimal solutions.
Revised: If we do not sample a sufﬁcient number of hyperparameters we may reach sub-optimal solutions.

Original: We approach this problem in section 2.4.
Revised: We approach this problem in section 2.4.

Original: An obvious difﬁculty of this approach is that training a hypernet typically requires training several times as many parameters as training a single model.
Revised: An obvious difﬁculty of this approach is that training a hypernet typically requires training several times as many parameters as training a single model.

Original: For example, training a fully-connected hypernet with a single hidden layer of H units to output D parameters requires training at least D
Revised: For example, training a fully-connected hypernet with a single hidden layer of H units to output D parameters requires training at least D

Original: × H hypernet parameters.
Revised: × H hypernet parameters.

Original: Again, in section 2.4 we propose an algorithm that requires training only a linear model mapping hyperparameters to model weights.
Revised: Again, in section 2.4 we propose an algorithm that requires training only a linear model mapping hyperparameters to model weights.

Original: 2. A SYMPTOTIC CONVERGENCE PROPERTIES
Revised: 2. A SYMPTOTIC CONVERGENCE PROPERTIES

Original: Algorithm 2 trains a hypernet using SGD, drawing hyperparameters from a ﬁxed distribution p ( λ ) .
Revised: Algorithm 2 trains a hypernet using SGD, drawing hyperparameters from a ﬁxed distribution p ( λ ) .

Original: This section proves that Algorithm 2 converges to a local best-response under mild assumptions.
Revised: This section proves that Algorithm 2 converges to a local best-response under mild assumptions.

Original: In particular, we show that, for a sufﬁciently large hypernet, the choice of p ( λ ) does not matter as long as it has sufﬁcient support.
Revised: In particular, we show that, for a sufﬁciently large hypernet, the choice of p ( λ ) does not matter as long as it has sufﬁcient support.

Original: Notation as if w φ has a unique solution for φ or w is used for simplicity, but is not true in general.
Revised: Notation as if w φ has a unique solution for φ or w is used for simplicity, but is not true in general.

Original: Theorem 2.1.
Revised: Theorem 2.1.

Original: Sufﬁciently powerful hypernets can learn continuous best-response functions, which minimizes the expected loss for any hyperparameter distribution.
Revised: Sufﬁciently powerful hypernets can learn continuous best-response functions, which minimizes the expected loss for any hyperparameter distribution.

Original: There exists φ ∗ , such that for all λ ∈ support( p ( λ )) ,
Revised: There exists φ ∗ , such that for all λ ∈ support( p ( λ )) ,

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: If w φ is a universal approximator (Hornik, 1991) and the best-response is continuous in λ (which allows approximation by w φ ), then there exists optimal hypernet parameters φ ∗ such that for all hyperparameters λ , w φ ∗ ( λ )
Revised: If w φ is a universal approximator (Hornik, 1991) and the best-response is continuous in λ (which allows approximation by w φ ), then there exists optimal hypernet parameters φ ∗ such that for all hyperparameters λ , w φ ∗ ( λ )

Original: =
Revised: =

Original: argmin w L Train (w , λ ) .
Revised: argmin w L Train (w , λ ) .

Original: Thus, L Train (w φ ∗ ( λ ) , λ ) =
Revised: Thus, L Train (w φ ∗ ( λ ) , λ ) =

Original: min w L Train (w , λ ) .
Revised: min w L Train (w , λ ) .

Original: In other words, universal approximator hypernets can learn continuous bestresponses.
Revised: In other words, universal approximator hypernets can learn continuous bestresponses.

Original: ) 4: φ = φ − α ∇ φ L Train ( x , w φ (ˆ λ ) , ˆ λ ) 5: λ λ − β ∇
Revised: ) 4: φ = φ − α ∇ φ L Train ( x , w φ (ˆ λ ) , ˆ λ ) 5: λ λ − β ∇

Original: 1: initialize φ, ˆ λ 2:
Revised: 1: initialize φ, ˆ λ 2:

Original: T joint steps 3: x ∼ Training data, λ ∼ p ( λ | ˆ λ ) 4: φ = φ − α ∇ φ L Train ( x , w φ (ˆ λ ) , ˆ λ ) 5: 6: x ∼ Validation data 7: ˆ λ
Revised: T joint steps 3: x ∼ Training data, λ ∼ p ( λ | ˆ λ ) 4: φ = φ − α ∇ φ L Train ( x , w φ (ˆ λ ) , ˆ λ ) 5: 6: x ∼ Validation data 7: ˆ λ

Original: =
Revised: =

Original: ˆ λ − β ∇ ˆ λ L Valid .
Revised: ˆ λ − β ∇ ˆ λ L Valid .

Original: ( x , w φ (ˆ λ )) 8: return ˆ λ, w φ (ˆ λ )
Revised: ( x , w φ (ˆ λ )) 8: return ˆ λ, w φ (ˆ λ )

Original: Substituting φ ∗ into the training loss gives E p ( λ ) [ L Train (w φ ∗ ( λ ) , λ )]
Revised: Substituting φ ∗ into the training loss gives E p ( λ ) [ L Train (w φ ∗ ( λ ) , λ )]

Original: =
Revised: =

Original: E p ( λ ) [min φ L Train (w φ ( λ ) , λ )] .
Revised: E p ( λ ) [min φ L Train (w φ ( λ ) , λ )] .

Original: By Jensen’s inequality, min φ E p ( λ )
Revised: By Jensen’s inequality, min φ E p ( λ )

Original: [ L Train (w φ ( λ ) , λ )] ≥ E p ( λ ) [min φ L Train (w φ ( λ ) , λ )] where min φ is a convex function on the convex vector space of functions {L Train (w φ ( λ ) , λ ) for λ ∈ support( p ( λ )) } if support( p ( λ )) is convex and L Train (w , λ ) =
Revised: [ L Train (w φ ( λ ) , λ )] ≥ E p ( λ ) [min φ L Train (w φ ( λ ) , λ )] where min φ is a convex function on the convex vector space of functions {L Train (w φ ( λ ) , λ ) for λ ∈ support( p ( λ )) } if support( p ( λ )) is convex and L Train (w , λ ) =

Original: E x ∼ Train
Revised: E x ∼ Train

Original: [ L Pred ( x , w)]
Revised: [ L Pred ( x , w)]

Original: +
Revised: +

Original: L Reg (w , λ ) with L Reg (w , λ ) =
Revised: L Reg (w , λ ) with L Reg (w , λ ) =

Original: λ · L (w) .
Revised: λ · L (w) .

Original: Thus, φ ∗ = argmin
Revised: Thus, φ ∗ = argmin

Original: φ E p ( λ ) [ L Train (w φ ( λ ) , λ )] .
Revised: φ E p ( λ ) [ L Train (w φ ( λ ) , λ )] .

Original: In other words, if the hypernet learns the best-response it will simultaneously minimize the loss for every point in the support( p ( λ )) .
Revised: In other words, if the hypernet learns the best-response it will simultaneously minimize the loss for every point in the support( p ( λ )) .

Original: Thus, having a universal approximator and a continuous best-response implies for all λ ∈ support( p ( λ )) , L Valid .
Revised: Thus, having a universal approximator and a continuous best-response implies for all λ ∈ support( p ( λ )) , L Valid .

Original: (w φ ∗ ( λ ))
Revised: (w φ ∗ ( λ ))

Original: =
Revised: =

Original: L Valid .
Revised: L Valid .

Original: (w ∗ ( λ )) because w φ ∗ ( λ ) =
Revised: (w ∗ ( λ )) because w φ ∗ ( λ ) =

Original: w ∗ ( λ ) .
Revised: w ∗ ( λ ) .

Original: Thus, under mild conditions, we will learn a best-response in the support of the hyperparameter distribution.
Revised: Thus, under mild conditions, we will learn a best-response in the support of the hyperparameter distribution.

Original: If the best-response is differentiable, then there is a neighborhood about each hyperparameter where the best-response is approximately linear.
Revised: If the best-response is differentiable, then there is a neighborhood about each hyperparameter where the best-response is approximately linear.

Original: If we select the support of the hyperparameter distribution to be the neighborhood where the best-response is approximately linear then we can use linear regression to learn the best-response locally.
Revised: If we select the support of the hyperparameter distribution to be the neighborhood where the best-response is approximately linear then we can use linear regression to learn the best-response locally.

Original: Theorem 2.1 holds for any p ( λ ) .
Revised: Theorem 2.1 holds for any p ( λ ) .

Original: However in practice, we have a limited-capacity hypernet, and so should choose a p ( λ ) that puts most of its mass on promising hyperparameter values.
Revised: However in practice, we have a limited-capacity hypernet, and so should choose a p ( λ ) that puts most of its mass on promising hyperparameter values.

Original: This motivates the joint optimization of φ and p ( λ ) .
Revised: This motivates the joint optimization of φ and p ( λ ) .

Original: Concretely, we can introduce a “current” hyperparameter ˆ λ and deﬁne a conditional hyperparameter distribution p ( λ | ˆ λ ) which places its mass near ˆ λ .
Revised: Concretely, we can introduce a “current” hyperparameter ˆ λ and deﬁne a conditional hyperparameter distribution p ( λ | ˆ λ ) which places its mass near ˆ λ .

Original: This allows us to use a limited-capacity hypernet, at the cost of having to re-train the hypernet each time we update ˆ λ .
Revised: This allows us to use a limited-capacity hypernet, at the cost of having to re-train the hypernet each time we update ˆ λ .

Original: In practice, there are no guarantees about the network being a universal approximator or the ﬁnitetime convergence of optimization.
Revised: In practice, there are no guarantees about the network being a universal approximator or the ﬁnitetime convergence of optimization.

Original: The optimal hypernet will depend on the hyperparameter distribution p ( λ ) , not just the support of this distribution.
Revised: The optimal hypernet will depend on the hyperparameter distribution p ( λ ) , not just the support of this distribution.

Original: We appeal to experimental results that our method is feasible in practice.
Revised: We appeal to experimental results that our method is feasible in practice.

Original: 2. J OINTLY TRAINING PARAMETERS AND HYPERPARAMETERS
Revised: 2. J OINTLY TRAINING PARAMETERS AND HYPERPARAMETERS

Original: Because in practice we use a limited-capacity hypernet, it may not be possible to learn a bestresponse for all hyperparameters.
Revised: Because in practice we use a limited-capacity hypernet, it may not be possible to learn a bestresponse for all hyperparameters.

Original: Thus, we propose Algorithm 3, which only tries to learn a bestresponse locally.
Revised: Thus, we propose Algorithm 3, which only tries to learn a bestresponse locally.

Original: We introduce a “current” hyperparameter ˆ λ , which is updated each iteration.
Revised: We introduce a “current” hyperparameter ˆ λ , which is updated each iteration.

Original: We deﬁne a conditional hyperparameter distribution, p ( λ | ˆ λ ) , which only puts mass close to ˆ λ .
Revised: We deﬁne a conditional hyperparameter distribution, p ( λ | ˆ λ ) , which only puts mass close to ˆ λ .

Original: Algorithm 3 combines the two phases of Algorithm 2 into one.
Revised: Algorithm 3 combines the two phases of Algorithm 2 into one.

Original: Instead of ﬁrst learning a hypernet that can output weights for any hyperparameter then optimizing the hyperparameters, Algorithmonly samples hyperparameters near the current best guess.
Revised: Instead of ﬁrst learning a hypernet that can output weights for any hyperparameter then optimizing the hyperparameters, Algorithmonly samples hyperparameters near the current best guess.

Original: This means the hypernet only has to be trained to estimate good enough weights for a small set of hyperparameters.
Revised: This means the hypernet only has to be trained to estimate good enough weights for a small set of hyperparameters.

Original: The locally-trained hypernet can then be used to provide gradients to update the hyperparameters based on validation set performance.
Revised: The locally-trained hypernet can then be used to provide gradients to update the hyperparameters based on validation set performance.

Original: How simple can we make the hypernet, and still obtain useful gradients to optimize hyperparameters?
Revised: How simple can we make the hypernet, and still obtain useful gradients to optimize hyperparameters?

Original: Consider the case used in our experiments where the hypernet is a linear function of the hyperparameters and the conditional hyperparameter distribution is p ( λ | ˆ λ )
Revised: Consider the case used in our experiments where the hypernet is a linear function of the hyperparameters and the conditional hyperparameter distribution is p ( λ | ˆ λ )

Original: =
Revised: =

Original: N (ˆ λ, σ 1 ) for some small σ .
Revised: N (ˆ λ, σ 1 ) for some small σ .

Original: This hypernet learns a tangent hyperplane to a best-response function and only needs to make small adjustments at each step if the hyperparameter updates are sufﬁciently small.
Revised: This hypernet learns a tangent hyperplane to a best-response function and only needs to make small adjustments at each step if the hyperparameter updates are sufﬁciently small.

Original: We can further restrict the capacity of a linear hypernet by factorizing its weights, effectively adding a bottleneck layer with a linear activation and a small number of hidden units.
Revised: We can further restrict the capacity of a linear hypernet by factorizing its weights, effectively adding a bottleneck layer with a linear activation and a small number of hidden units.

Original: [Figure]
Revised: [Figure]

Original: L
Revised: L

Original: [Figure]
Revised: [Figure]

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Our work is closely related to the concurrent work of
Revised: Our work is closely related to the concurrent work of

Original: Brock et al.
Revised: Brock et al.

Original: (2017), whose SMASH algorithm also approximates the optimal weights as a function of model architectures, to perform a gradient-free search over discrete model structures.
Revised: (2017), whose SMASH algorithm also approximates the optimal weights as a function of model architectures, to perform a gradient-free search over discrete model structures.

Original: Their work focuses on efﬁciently evaluating the performance of a variety of discrete model architectures, while we focus on efﬁciently exploring continuous spaces of models.
Revised: Their work focuses on efﬁciently evaluating the performance of a variety of discrete model architectures, while we focus on efﬁciently exploring continuous spaces of models.

Original: Model-free approaches Model-free approaches only use trial-and-error to explore the hyperparameter space.
Revised: Model-free approaches Model-free approaches only use trial-and-error to explore the hyperparameter space.

Original: Simple model-free approaches applied to hyperparameter optimization include grid search and random search (Bergstra & Bengio, 2012).
Revised: Simple model-free approaches applied to hyperparameter optimization include grid search and random search (Bergstra & Bengio, 2012).

Original: Hyperband (Li et al., 2016) combines bandit approaches with modeling the learning procedure.
Revised: Hyperband (Li et al., 2016) combines bandit approaches with modeling the learning procedure.

Original: Model-based approaches Model-based approaches try to build a surrogate function, which often allows gradient-based optimization or active learning.
Revised: Model-based approaches Model-based approaches try to build a surrogate function, which often allows gradient-based optimization or active learning.

Original: A common example is Bayesian optimization (Snoek et al., 2012).
Revised: A common example is Bayesian optimization (Snoek et al., 2012).

Original: Freeze-thaw Bayesian optimization can condition on partially-optimized model performance.
Revised: Freeze-thaw Bayesian optimization can condition on partially-optimized model performance.

Original: Optimization-based approaches Another line of related work attempts to directly approximate gradients of the validation loss with respect to hyperparameters.
Revised: Optimization-based approaches Another line of related work attempts to directly approximate gradients of the validation loss with respect to hyperparameters.

Original: Domke (2012) proposes to differentiate through unrolled optimization to approximate best-responses in nested optimization and Maclaurin et al.
Revised: Domke (2012) proposes to differentiate through unrolled optimization to approximate best-responses in nested optimization and Maclaurin et al.

Original: (2015a) differentiate through entire unrolled learning procedures.
Revised: (2015a) differentiate through entire unrolled learning procedures.

Original: DrMAD (Fu et al., 2016) approximates differentiating through an unrolled learning procedure to relax memory requirements for deep neural networks.
Revised: DrMAD (Fu et al., 2016) approximates differentiating through an unrolled learning procedure to relax memory requirements for deep neural networks.

Original: HOAG (Pedregosa, 2016) ﬁnds hyperparameter gradients with implicit differentiation by deriving an implicit equation for the gradient with optimality conditions.
Revised: HOAG (Pedregosa, 2016) ﬁnds hyperparameter gradients with implicit differentiation by deriving an implicit equation for the gradient with optimality conditions.

Original: Franceschi et al.
Revised: Franceschi et al.

Original: (2017) study forward and reverse-mode differentiation for constructing hyperparameter gradients.
Revised: (2017) study forward and reverse-mode differentiation for constructing hyperparameter gradients.

Original: Also, Feng & Simon (2017) establish conditions where the validation loss of best-responding weights are almost everywhere smooth, allowing gradient-based training of hyperparameters.
Revised: Also, Feng & Simon (2017) establish conditions where the validation loss of best-responding weights are almost everywhere smooth, allowing gradient-based training of hyperparameters.

Original: A closely-related procedure to our method is the T 1 − T 2 method of Luketina et al.
Revised: A closely-related procedure to our method is the T 1 − T 2 method of Luketina et al.

Original: (2016), which also provides an algorithm for stochastic gradient-based optimization of hyperparameters.
Revised: (2016), which also provides an algorithm for stochastic gradient-based optimization of hyperparameters.

Original: The convergence of their procedure to local optima of the validation loss depends on approximating the
Revised: The convergence of their procedure to local optima of the validation loss depends on approximating the

Original: Hessian of the training loss with respect to parameters with the identity matrix.
Revised: Hessian of the training loss with respect to parameters with the identity matrix.

Original: In contrast, the convergence of our method depends on having a suitably powerful hypernet.
Revised: In contrast, the convergence of our method depends on having a suitably powerful hypernet.

Original: Game theory Best-response functions are extensively studied as a solution concept in discrete and continuous multi-agent games (Fudenberg & Levine, 1998).
Revised: Game theory Best-response functions are extensively studied as a solution concept in discrete and continuous multi-agent games (Fudenberg & Levine, 1998).

Original: Games where learning a bestresponse can be applied include adversarial training (Goodfellow et al., 2014), or Stackelberg competitions (Br¨uckner & Scheffer, 2011).
Revised: Games where learning a bestresponse can be applied include adversarial training (Goodfellow et al., 2014), or Stackelberg competitions (Br¨uckner & Scheffer, 2011).

Original: For adversarial training, the analog of our method would be a discriminator which is trained while observing all of the generator’s parameters.
Revised: For adversarial training, the analog of our method would be a discriminator which is trained while observing all of the generator’s parameters.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In our experiments, we examine the standard example of stochastic gradient-based optimization of neural networks, with a weight regularization penalty.
Revised: In our experiments, we examine the standard example of stochastic gradient-based optimization of neural networks, with a weight regularization penalty.

Original: Gradient-based methods explicitly use the gradient of a loss, while gradient-free methods do not (but can use the gradient of a surrogate loss that is learned).
Revised: Gradient-based methods explicitly use the gradient of a loss, while gradient-free methods do not (but can use the gradient of a surrogate loss that is learned).

Original: Our algorithm may be contrasted with gradient-free techniques like Bayesian optimization, gradient-based methods only handle hyperparameters affecting the training loss and gradient-based methods which can additionally handle optimization parameters.
Revised: Our algorithm may be contrasted with gradient-free techniques like Bayesian optimization, gradient-based methods only handle hyperparameters affecting the training loss and gradient-based methods which can additionally handle optimization parameters.

Original: The best comparison for hyper-training is to gradient-based methods which only handle parameters affecting the training loss because other methods apply to a more general set of problems.
Revised: The best comparison for hyper-training is to gradient-based methods which only handle parameters affecting the training loss because other methods apply to a more general set of problems.

Original: In this case, the training and validation losses can be written as:
Revised: In this case, the training and validation losses can be written as:

Original: [Equation]
Revised: [Equation]

Original: In all experiments, Algorithms 2 or 3 are used to optimize weights of with mean squared error on
Revised: In all experiments, Algorithms 2 or 3 are used to optimize weights of with mean squared error on

Original: MNIST (LeCun et al., 1998) with L Reg as an L 2 weight decay penalty weighted by exp( λ ) .
Revised: MNIST (LeCun et al., 1998) with L Reg as an L 2 weight decay penalty weighted by exp( λ ) .

Original: The elementary model has 7 , 850 weights.
Revised: The elementary model has 7 , 850 weights.

Original: All hidden units in the hypernet have a ReLU activation (Nair & Hinton, 2010) unless otherwise speciﬁed.
Revised: All hidden units in the hypernet have a ReLU activation (Nair & Hinton, 2010) unless otherwise speciﬁed.

Original: Autograd (Maclaurin et al., 2015b) was used to compute all derivatives.
Revised: Autograd (Maclaurin et al., 2015b) was used to compute all derivatives.

Original: For each experiment, the minibatch samples 2 pairs of hyperparameters and up to 1 , 000 training data points.
Revised: For each experiment, the minibatch samples 2 pairs of hyperparameters and up to 1 , 000 training data points.

Original: Adam was used for training the hypernet and hyperparameters, with a step size of 0 . 0001 .
Revised: Adam was used for training the hypernet and hyperparameters, with a step size of 0 . 0001 .

Original: All experiments were run on a 2012 MacBook pro.
Revised: All experiments were run on a 2012 MacBook pro.

Original: 4. L EARNING A GLOBAL BEST - RESPONSE
Revised: 4. L EARNING A GLOBAL BEST - RESPONSE

Original: Our ﬁrst experiment, shown in Figure 1, demonstrates learning a global approximation to a bestresponse function using Algorithm 2.
Revised: Our ﬁrst experiment, shown in Figure 1, demonstrates learning a global approximation to a bestresponse function using Algorithm 2.

Original: To make visualization of the regularization loss easier, we use 10 training data points to exacerbate overﬁtting.
Revised: To make visualization of the regularization loss easier, we use 10 training data points to exacerbate overﬁtting.

Original: We compare the performance of weights output by the hypernet to those trained by standard cross-validation (Algorithm 1).
Revised: We compare the performance of weights output by the hypernet to those trained by standard cross-validation (Algorithm 1).

Original: Thus, elementary weights were randomly initialized for each hyperparameter setting and optimized using Adam (Kingma &
Revised: Thus, elementary weights were randomly initialized for each hyperparameter setting and optimized using Adam (Kingma &

Original: Ba, 2014) for 1 , 000 iterations with a step size of 0 . 0001 .
Revised: Ba, 2014) for 1 , 000 iterations with a step size of 0 . 0001 .

Original: When training the hypernetwork, hyperparameters were sampled from a broad Gaussian distribution: p ( λ ) =
Revised: When training the hypernetwork, hyperparameters were sampled from a broad Gaussian distribution: p ( λ ) =

Original: N (0 , 1 . 5) .
Revised: N (0 , 1 . 5) .

Original: The hypernet has 50 hidden units which results in 400 , 450 parameters of the hypernetwork.
Revised: The hypernet has 50 hidden units which results in 400 , 450 parameters of the hypernetwork.

Original: The minimum of the best-response in Figure 1 is close to the true minimum of the validation loss, which shows a hypernet can satisfactorily approximate a global best-response function in small problems.
Revised: The minimum of the best-response in Figure 1 is close to the true minimum of the validation loss, which shows a hypernet can satisfactorily approximate a global best-response function in small problems.

Original: 4. L EARNING A LOCAL BEST - RESPONSE
Revised: 4. L EARNING A LOCAL BEST - RESPONSE

Original: Figure 5 shows the same experiment, but using the fused updates of Algorithm 3.
Revised: Figure 5 shows the same experiment, but using the fused updates of Algorithm 3.

Original: This results in ﬁnding a best-response approximation whose minimum is the true minimum faster than the prior experiment.
Revised: This results in ﬁnding a best-response approximation whose minimum is the true minimum faster than the prior experiment.

Original: The conditional hyperparameter distribution is given by p ( λ | ˆ λ ) =
Revised: The conditional hyperparameter distribution is given by p ( λ | ˆ λ ) =

Original: N (ˆ λ, 0 . 00001) .
Revised: N (ˆ λ, 0 . 00001) .

Original: The hypernet is a linear model, with only 15 , 700 weights.
Revised: The hypernet is a linear model, with only 15 , 700 weights.

Original: We use the same optimizer as the global best-response to update both the hypernet and the hyperparameters.
Revised: We use the same optimizer as the global best-response to update both the hypernet and the hyperparameters.

Original: Again, the minimum of the best-response at the end of training is the true optimum on the validation loss.
Revised: Again, the minimum of the best-response at the end of training is the true optimum on the validation loss.

Original: This experiment shows that using only a locally-trained linear best-response function can give sufﬁcient gradient information to optimize hyperparameters on a small problem.
Revised: This experiment shows that using only a locally-trained linear best-response function can give sufﬁcient gradient information to optimize hyperparameters on a small problem.

Original: Algorithm 3 is also less computationally expensive than Algorithms 1 or 2.
Revised: Algorithm 3 is also less computationally expensive than Algorithms 1 or 2.

Original: 4. H YPER - TRAINING AND UNROLLED OPTIMIZATION
Revised: 4. H YPER - TRAINING AND UNROLLED OPTIMIZATION

Original: In order to compare hyper-training with other gradient-based hyperparameter optimization we trained a model with 7 , 850 hyperparameters with a separate L 2 weight decay applied to each weight in a 1 layer (linear) model.
Revised: In order to compare hyper-training with other gradient-based hyperparameter optimization we trained a model with 7 , 850 hyperparameters with a separate L 2 weight decay applied to each weight in a 1 layer (linear) model.

Original: The conditional hyperparameter distribution and optimizer for the hypernet and hyperparameters is the same the prior experiment.
Revised: The conditional hyperparameter distribution and optimizer for the hypernet and hyperparameters is the same the prior experiment.

Original: The weights for the model are factorized by selecting a hypernet with 10 hidden units.
Revised: The weights for the model are factorized by selecting a hypernet with 10 hidden units.

Original: The factorized linear hypernet has 10 hidden units giving 164 , 860 weights.
Revised: The factorized linear hypernet has 10 hidden units giving 164 , 860 weights.

Original: This means each hypernet iteration 2 · 10 times as expensive as an iteration on just the model, because there is the same number of hyperparameters as model parameters.
Revised: This means each hypernet iteration 2 · 10 times as expensive as an iteration on just the model, because there is the same number of hyperparameters as model parameters.

Original: [Figure]
Revised: [Figure]

Original: with

Original: [Figure]
Revised: [Figure]

Original: Figure 6, left, shows that Algorithm 3 converges more quickly than the unrolled reverse-mode optimization introduced in Maclaurin et al.
Revised: Figure 6, left, shows that Algorithm 3 converges more quickly than the unrolled reverse-mode optimization introduced in Maclaurin et al.

Original: (2015a)
Revised: (2015a)

Original: with an implementation by Franceschi et al.
Revised: with an implementation by Franceschi et al.

Original: Hyper-training reaches sub-optimal solutions because of limitations on how many hyperparameters can be sampled for each update but overﬁts validation data less than unrolling.
Revised: Hyper-training reaches sub-optimal solutions because of limitations on how many hyperparameters can be sampled for each update but overﬁts validation data less than unrolling.

Original: Standard Bayesian optimization cannot be scaled to this many hyperparameters.
Revised: Standard Bayesian optimization cannot be scaled to this many hyperparameters.

Original: Thus, this experiment shows Algorithm 3 can effectively partially optimize thousands of hyperparameters.
Revised: Thus, this experiment shows Algorithm 3 can effectively partially optimize thousands of hyperparameters.

Original: 4. O PTIMIZING WITH DEEPER NETWORKS
Revised: 4. O PTIMIZING WITH DEEPER NETWORKS

Original: In order to to see if we can optimize deeper networks with hyper-training we optimize models with 1, 2, and 3 layers with a separate L 2 weight decay applied to each weight.
Revised: In order to to see if we can optimize deeper networks with hyper-training we optimize models with 1, 2, and 3 layers with a separate L 2 weight decay applied to each weight.

Original: The conditional hyperparameter distribution and optimizer for the hypernet and hyperparameters is the same the prior experiment.
Revised: The conditional hyperparameter distribution and optimizer for the hypernet and hyperparameters is the same the prior experiment.

Original: The weights for each model are factorized by selecting a hypernet with 10 hidden units.
Revised: The weights for each model are factorized by selecting a hypernet with 10 hidden units.

Original: Again, standard Bayesian optimization cannot be scaled to this many hyperparameters.
Revised: Again, standard Bayesian optimization cannot be scaled to this many hyperparameters.

Original: Figure 6, right, shows that Algorithm 3 can scale to networks with multiple hidden layers and outperform hand-tuned settings.
Revised: Figure 6, right, shows that Algorithm 3 can scale to networks with multiple hidden layers and outperform hand-tuned settings.

Original: As more layers are added the difference between validation loss and testing loss decreases, and the model performs better on the validation set.
Revised: As more layers are added the difference between validation loss and testing loss decreases, and the model performs better on the validation set.

Original: Future work should compare other architectures like recurrent or convolutional networks.
Revised: Future work should compare other architectures like recurrent or convolutional networks.

Original: Additionally, note that more layers perform with lower training (not shown), validation, and test losses, as opposed to lower training loss but higher validation or test loss.
Revised: Additionally, note that more layers perform with lower training (not shown), validation, and test losses, as opposed to lower training loss but higher validation or test loss.

Original: This indicates that using weight decay on each weight could be a prior for generalization, or that hyper-training enforces another useful prior like the continuity of a best-response.
Revised: This indicates that using weight decay on each weight could be a prior for generalization, or that hyper-training enforces another useful prior like the continuity of a best-response.

Original: 4. E STIMATING WEIGHTS VERSUS ESTIMATING LOSS
Revised: 4. E STIMATING WEIGHTS VERSUS ESTIMATING LOSS

Original: As mentioned above, our approach differs from Bayesian optimization in that we try to learn to predict optimal weights, while Bayesian optimization attempts to directly model the validation loss of optimized weights.
Revised: As mentioned above, our approach differs from Bayesian optimization in that we try to learn to predict optimal weights, while Bayesian optimization attempts to directly model the validation loss of optimized weights.

Original: In this ﬁnal experiment, we untangle the reason for the better performance of our method: Is it because of a better inductive bias, or because our method can see more hyperparameter settings during optimization?
Revised: In this ﬁnal experiment, we untangle the reason for the better performance of our method: Is it because of a better inductive bias, or because our method can see more hyperparameter settings during optimization?

Original: First, we constructed a hyper-training set: We optimized 25 sets of weights to completion, given randomly-sampled hyperparameters.
Revised: First, we constructed a hyper-training set: We optimized 25 sets of weights to completion, given randomly-sampled hyperparameters.

Original: We chose 25 samples since that is the regime in which we expect Gaussian process-based approaches to have the largest advantage.
Revised: We chose 25 samples since that is the regime in which we expect Gaussian process-based approaches to have the largest advantage.

Original: We also constructed a validation set of 10 , 215 (optimized weight, hyperparameter) tuples generated in the same manner.
Revised: We also constructed a validation set of 10 , 215 (optimized weight, hyperparameter) tuples generated in the same manner.

Original: We then ﬁt a Gaussian process (GP) regression model with an RBF kernel from sklearn on the hyper-training data.
Revised: We then ﬁt a Gaussian process (GP) regression model with an RBF kernel from sklearn on the hyper-training data.

Original: A hypernet is ﬁt to the same dataset.
Revised: A hypernet is ﬁt to the same dataset.

Original: However, this hypernet was trained to
Revised: However, this hypernet was trained to

Original: [Figure]
Revised: [Figure]

Original: loss.
Revised: loss.

Original: ﬁt optimized training weights, not optimized validation loss.
Revised: ﬁt optimized training weights, not optimized validation loss.

Original: Finally, we optimize another hypernet using Algorithm 2, for the same amount of time as building the hyper-training set.
Revised: Finally, we optimize another hypernet using Algorithm 2, for the same amount of time as building the hyper-training set.

Original: The two hypernets were linear models and were trained with the same optimizer parameters as the 7 , 850 -dimensional hyperparameter optimization.
Revised: The two hypernets were linear models and were trained with the same optimizer parameters as the 7 , 850 -dimensional hyperparameter optimization.

Original: Figure 7 shows the distribution of prediction errors of these three models.
Revised: Figure 7 shows the distribution of prediction errors of these three models.

Original: We can see that the
Revised: We can see that the

Original: Gaussian process tends to underestimate loss.
Revised: Gaussian process tends to underestimate loss.

Original: The hypernet trained with the same small ﬁxed set of examples tends to overestimate loss.
Revised: The hypernet trained with the same small ﬁxed set of examples tends to overestimate loss.

Original: We conjecture that this is due to the hypernetwork producing bad weights in regions where it doesn’t have enough training data.
Revised: We conjecture that this is due to the hypernetwork producing bad weights in regions where it doesn’t have enough training data.

Original: Because the hypernet must provide actual weights to predict the validation loss, poorly-ﬁt regions will overestimate the validation loss.
Revised: Because the hypernet must provide actual weights to predict the validation loss, poorly-ﬁt regions will overestimate the validation loss.

Original: Finally, the hypernet trained with Algorithm 2 produces errors tightly centered around 0.
Revised: Finally, the hypernet trained with Algorithm 2 produces errors tightly centered around 0.

Original: The main takeaway from this experiment is a hypernet can learn more accurate surrogate functions than a GP for equal compute budgets because it views (noisy) evaluations of more points.
Revised: The main takeaway from this experiment is a hypernet can learn more accurate surrogate functions than a GP for equal compute budgets because it views (noisy) evaluations of more points.

Original: Code for all experiments will be made available upon publication.
Revised: Code for all experiments will be made available upon publication.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: In this paper, we:
Revised: In this paper, we:

Original: • Presented algorithms that efﬁciently learn a differentiable approximation to a best-response without nested optimization.
Revised: • Presented algorithms that efﬁciently learn a differentiable approximation to a best-response without nested optimization.

Original: • Showed empirically that hypernets can provide a better inductive bias for hyperparameter optimization than Gaussian processes ﬁt directly to the validation loss.
Revised: • Showed empirically that hypernets can provide a better inductive bias for hyperparameter optimization than Gaussian processes ﬁt directly to the validation loss.

Original: • Gave a theoretical justiﬁcation that sufﬁciently large networks will learn the best-response for all hyperparameters it is trained against.
Revised: • Gave a theoretical justiﬁcation that sufﬁciently large networks will learn the best-response for all hyperparameters it is trained against.

Original: We hope that this initial exploration of stochastic hyperparameter optimization will inspire further reﬁnements, such as hyper-regularization methods, or uncertainty-aware exploration using Bayesian hypernetworks.
Revised: We hope that this initial exploration of stochastic hyperparameter optimization will inspire further reﬁnements, such as hyper-regularization methods, or uncertainty-aware exploration using Bayesian hypernetworks.

Original: A E XTRA E XPERIMENTS A. O PTIMIZING 10 H YPERPARAMETERS
Revised: A E XTRA E XPERIMENTS A. O PTIMIZING 10 H YPERPARAMETERS

Original: Here, we optimize a model with 10 hyperparameters, in which a separate L 2 weight decay is applied to the weights for each digit class in a linear regression model to see if we can optimize medium-sized models.
Revised: Here, we optimize a model with 10 hyperparameters, in which a separate L 2 weight decay is applied to the weights for each digit class in a linear regression model to see if we can optimize medium-sized models.

Original: The conditional hyperparameter distribution and optimizer for the hypernet and hyperparameters is the same the prior experiments.
Revised: The conditional hyperparameter distribution and optimizer for the hypernet and hyperparameters is the same the prior experiments.

Original: A linear hypernet is used, resulting in 86 , 350 hyper-weights.
Revised: A linear hypernet is used, resulting in 86 , 350 hyper-weights.

Original: Algorithm 3 is compared against random search and .
Revised: Algorithm 3 is compared against random search and .

Original: Figure 8, right
Revised: Figure 8, right

Original: [Figure]
Revised: [Figure]

Original: shows that our method converges more quickly and to a better optimum than either alternative method, demonstrating that medium-sized hyperparameter optimization problems can be solved with Algorithm
Revised: shows that our method converges more quickly and to a better optimum than either alternative method, demonstrating that medium-sized hyperparameter optimization problems can be solved with Algorithm

Original: [Figure]
Revised: [Figure]

Original: the best result of a random search.
Revised: the best result of a random search.

Original: Hypernetwork-based optimization converges faster than random search or Bayesian optimization.
Revised: Hypernetwork-based optimization converges faster than random search or Bayesian optimization.

Original: We also observe signiﬁcant overﬁtting of the hyperparameters on the validation set, which may be reduced be introducing hyperhyperparameters (parameters of the hyperparameter prior).
Revised: We also observe signiﬁcant overﬁtting of the hyperparameters on the validation set, which may be reduced be introducing hyperhyperparameters (parameters of the hyperparameter prior).

Original: The runtime includes the inner optimization for gradient-free approaches so that equal cumulative computational time is compared for each method.
Revised: The runtime includes the inner optimization for gradient-free approaches so that equal cumulative computational time is compared for each method.

Original: Factors affecting this include removing the overhead of constructing tuples of hyperparameters and optimized weights, viewing more hyperparameter samples, or having a better inductive bias from learning weights.
Revised: Factors affecting this include removing the overhead of constructing tuples of hyperparameters and optimized weights, viewing more hyperparameter samples, or having a better inductive bias from learning weights.