Original: F EW -S HOT

Original: B AYESIAN

Original:  O PTIMIZATION WITH D EEP K ERNEL S URROGATES
Revised: F EW -S HOT B AYESIAN O PTIMIZATION WITH D EEP K ERNEL S URROGATES

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Hyperparameter optimization (HPO) is a central pillar in the automation of machine learning solutions and is mainly performed via Bayesian optimization, where a parametric surrogate is learned to approximate the black box response function (e.g. validation error).
Revised: Hyperparameter optimization (HPO) is a central pillar in the automation of machine learning solutions and is mainly performed via Bayesian optimization, where a parametric surrogate is learned to approximate the black box response function (e.g. validation error).

Original: Unfortunately, evaluating the response function is computationally intensive.
Revised: Unfortunately, evaluating the response function is computationally intensive.

Original: As a remedy, earlier work emphasizes the need for transfer learning surrogates which learn to optimize hyperparameters for an algorithm from other tasks.
Revised: As a remedy, earlier work emphasizes the need for transfer learning surrogates which learn to optimize hyperparameters for an algorithm from other tasks.

Original: In contrast to previous work, we propose to rethink HPO as a few-shot learning problem in which we train a shared deep surrogate model to quickly adapt (with few response evaluations) to the response function of a new task.
Revised: In contrast to previous work, we propose to rethink HPO as a few-shot learning problem in which we train a shared deep surrogate model to quickly adapt (with few response evaluations) to the response function of a new task.

Original: We propose the use of a deep kernel network for a Gaussian process surrogate that is meta-learned in an end-to-end fashion in order to jointly approximate the response functions of a collection of training data sets.
Revised: We propose the use of a deep kernel network for a Gaussian process surrogate that is meta-learned in an end-to-end fashion in order to jointly approximate the response functions of a collection of training data sets.

Original: As a result, the novel few-shot optimization of our deep kernel surrogate leads to new state-of-the-art results at HPO compared to several recent methods on diverse metadata sets.
Revised: As a result, the novel few-shot optimization of our deep kernel surrogate leads to new state-of-the-art results at HPO compared to several recent methods on diverse metadata sets.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Many machine learning models have very sensitive hyperparameters that must be carefully tuned for efﬁcient use.
Revised: Many machine learning models have very sensitive hyperparameters that must be carefully tuned for efﬁcient use.

Original: Unfortunately, ﬁnding the right setting is a tedious trial and error process that requires expert knowledge.
Revised: Unfortunately, ﬁnding the right setting is a tedious trial and error process that requires expert knowledge.

Original: AutoML methods address this problem by providing tools to automate hyperparameter optimization, where Bayesian optimization has become the standard for this task (Snoek et al., 2012).
Revised: AutoML methods address this problem by providing tools to automate hyperparameter optimization, where Bayesian optimization has become the standard for this task (Snoek et al., 2012).

Original: It treats the problem of hyperparameter optimization as a black box optimization problem.
Revised: It treats the problem of hyperparameter optimization as a black box optimization problem.

Original: Here the black box function is the hyperparameter response function, which maps a hyperparameter setting to the validation loss.
Revised: Here the black box function is the hyperparameter response function, which maps a hyperparameter setting to the validation loss.

Original: Bayesian optimization consists of two parts.
Revised: Bayesian optimization consists of two parts.

Original: First, a surrogate model, often a Gaussian process, is used to approximate the response function.
Revised: First, a surrogate model, often a Gaussian process, is used to approximate the response function.

Original: Second, an acquisition function is used that balances the trade-off between exploration and exploitation.
Revised: Second, an acquisition function is used that balances the trade-off between exploration and exploitation.

Original: In a sequential process, hyperparameter settings are selected and evaluated, followed by an update of the surrogate model.
Revised: In a sequential process, hyperparameter settings are selected and evaluated, followed by an update of the surrogate model.

Original: Recently, several attempts have been made to extend Bayesian optimization to account for a transfer learning setup.
Revised: Recently, several attempts have been made to extend Bayesian optimization to account for a transfer learning setup.

Original: It is assumed here that historical information on machine learning algorithms is available with different hyperparameters.
Revised: It is assumed here that historical information on machine learning algorithms is available with different hyperparameters.

Original: This can either be because this information is publicly available (e.g. OpenML) or because the algorithm is repeatedly optimized for different data sets.
Revised: This can either be because this information is publicly available (e.g. OpenML) or because the algorithm is repeatedly optimized for different data sets.

Original: To this end, several transfer learning surrogates have been proposed that use this additional information to reduce the convergence time of Bayesian optimization.
Revised: To this end, several transfer learning surrogates have been proposed that use this additional information to reduce the convergence time of Bayesian optimization.

Original: We propose a new paradigm for accomplishing the knowledge transfer by reconceptualizing the process as a few-shot learning task.
Revised: We propose a new paradigm for accomplishing the knowledge transfer by reconceptualizing the process as a few-shot learning task.

Original: Inspiration is drawn from the fact that there are a limited number of black box function evaluations for a new hyperparameter optimization task (i.e. few shots) but there are ample evaluations of related black box objectives (i.e. evaluated hyperparameters on other data sets).
Revised: Inspiration is drawn from the fact that there are a limited number of black box function evaluations for a new hyperparameter optimization task (i.e. few shots) but there are ample evaluations of related black box objectives (i.e. evaluated hyperparameters on other data sets).

Original: This approach has several advantages.
Revised: This approach has several advantages.

Original: First, a single model is learned that is trained to quickly adapt to a new task when few examples are available.
Revised: First, a single model is learned that is trained to quickly adapt to a new task when few examples are available.

Original: This is exactly the challenge we face when optimizing hyperparameters.
Revised: This is exactly the challenge we face when optimizing hyperparameters.

Original: Second, this method can be scaled very well for any number of considered tasks.
Revised: Second, this method can be scaled very well for any number of considered tasks.

Original: This not only enables the learning from large metadata sets but also enables the problem of label normalization to be dealt with in a new way.
Revised: This not only enables the learning from large metadata sets but also enables the problem of label normalization to be dealt with in a new way.

Original: Finally, we present an evolutionary algorithm that can use surrogate models to get a warm start initialization for Bayesian optimization.
Revised: Finally, we present an evolutionary algorithm that can use surrogate models to get a warm start initialization for Bayesian optimization.

Original: All of our contributions are empirically compared with several competitive methods in three different problems.
Revised: All of our contributions are empirically compared with several competitive methods in three different problems.

Original: Two ablation studies provide information about the inﬂuence of the individual components.
Revised: Two ablation studies provide information about the inﬂuence of the individual components.

Original: Concluding, our contributions in this work are:
Revised: Concluding, our contributions in this work are:

Original: • This is the ﬁrst work that, in the context of hyperparameter optimization (HPO), trains the initialization of the parameters of a probabilistic surrogate model from a collection of meta-tasks by few-shot learning and then transfers it by ﬁne-tuning the initialized kernel parameters to a target task.
Revised: • This is the ﬁrst work that, in the context of hyperparameter optimization (HPO), trains the initialization of the parameters of a probabilistic surrogate model from a collection of meta-tasks by few-shot learning and then transfers it by ﬁne-tuning the initialized kernel parameters to a target task.

Original: • We are the ﬁrst to consider transfer learning in HPO as a few-shot learning task.
Revised: • We are the ﬁrst to consider transfer learning in HPO as a few-shot learning task.

Original: • We set the new state of the art in transfer learning for the HPO and provide ample evidence that we outperform strong baselines published at ICLR and NeurIPS with a statistically signiﬁcant margin.
Revised: • We set the new state of the art in transfer learning for the HPO and provide ample evidence that we outperform strong baselines published at ICLR and NeurIPS with a statistically signiﬁcant margin.

Original: • We present an evolutionary algorithm that can use surrogate models to get a warm start initialization for Bayesian optimization.
Revised: • We present an evolutionary algorithm that can use surrogate models to get a warm start initialization for Bayesian optimization.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: The idea of using transfer learning to improve Bayesian optimization is being investigated in several papers.
Revised: The idea of using transfer learning to improve Bayesian optimization is being investigated in several papers.

Original: The early work suggests learning a single Gaussian process for the entire data (Bardenet et al., 2013; Swersky et al., 2013; Yogatama & Mann, 2014).
Revised: The early work suggests learning a single Gaussian process for the entire data (Bardenet et al., 2013; Swersky et al., 2013; Yogatama & Mann, 2014).

Original: Since the training of a Gaussian process is cubic in the number of training points, this idea does not scale well.
Revised: Since the training of a Gaussian process is cubic in the number of training points, this idea does not scale well.

Original: This problem has recently been addressed by several proposals to use ensembles of Gaussian processes where a Gaussian process is learned for each task (Wistuba et al., 2018; Feurer et al., 2018).
Revised: This problem has recently been addressed by several proposals to use ensembles of Gaussian processes where a Gaussian process is learned for each task (Wistuba et al., 2018; Feurer et al., 2018).

Original: This idea scales linearly in the number of tasks but still cubically in the number of training points per task.
Revised: This idea scales linearly in the number of tasks but still cubically in the number of training points per task.

Original: Thus, the problem persists in scenarios where there is a lot of data available for each task.
Revised: Thus, the problem persists in scenarios where there is a lot of data available for each task.

Original: Bayesian neural networks are a possible more scalable way of learning with large amounts of data.
Revised: Bayesian neural networks are a possible more scalable way of learning with large amounts of data.

Original: For example, Schilling et al.
Revised: For example, Schilling et al.

Original: (2015) propose to use a neural network with task embedding and variable interactions.
Revised: (2015) propose to use a neural network with task embedding and variable interactions.

Original: To obtain mean and variance predictions, the authors propose using an ensemble of models.
Revised: To obtain mean and variance predictions, the authors propose using an ensemble of models.

Original: In contrast, Springenberg et al.
Revised: In contrast, Springenberg et al.

Original: (2016) use a Bayesian multi-task neural network.
Revised: (2016) use a Bayesian multi-task neural network.

Original: However, since training Bayesian neural networks is computationally intensive, Perrone et al.
Revised: However, since training Bayesian neural networks is computationally intensive, Perrone et al.

Original: (2018) propose a more scalable approach.
Revised: (2018) propose a more scalable approach.

Original: They suggest using a neural network that is shared by all tasks and using Bayesian linear regression for each task.
Revised: They suggest using a neural network that is shared by all tasks and using Bayesian linear regression for each task.

Original: The parameters are trained jointly on the entire data.
Revised: The parameters are trained jointly on the entire data.

Original: While our work shares some similarities with the previous work, our algorithm has unique properties.
Revised: While our work shares some similarities with the previous work, our algorithm has unique properties.

Original: First of all, a meta-learning algorithm is used, which is motivated by recent work on model-agnostic meta-learning for few-shot learning (Finn et al., 2017).
Revised: First of all, a meta-learning algorithm is used, which is motivated by recent work on model-agnostic meta-learning for few-shot learning (Finn et al., 2017).

Original: This will allow us to integrate all task-speciﬁc parameters out such that the model does not grow with the number of tasks.
Revised: This will allow us to integrate all task-speciﬁc parameters out such that the model does not grow with the number of tasks.

Original: As a result, our algorithm scales very well with the number of tasks.
Revised: As a result, our algorithm scales very well with the number of tasks.

Original: Second, while we are also using a neural network, we combine it with a Gaussian process with a nonlinear kernel in order to obtain uncertainty predictions.
Revised: Second, while we are also using a neural network, we combine it with a Gaussian process with a nonlinear kernel in order to obtain uncertainty predictions.

Original: A simpler solution for using the transfer learning idea in Bayesian optimization is initialization (Feurer et al., 2015; Wistuba et al., 2015a).
Revised: A simpler solution for using the transfer learning idea in Bayesian optimization is initialization (Feurer et al., 2015; Wistuba et al., 2015a).

Original: The standard Bayesian optimization routine with a simple Gaussian process is used in this case but it is warm-started by a number of hyperparameter settings that work well for related tasks.
Revised: The standard Bayesian optimization routine with a simple Gaussian process is used in this case but it is warm-started by a number of hyperparameter settings that work well for related tasks.

Original: We also explore this idea in the context of this paper by proposing a simple evolutionary algorithm that can use a surrogate model to estimate a data-driven warm start initialization sequence.
Revised: We also explore this idea in the context of this paper by proposing a simple evolutionary algorithm that can use a surrogate model to estimate a data-driven warm start initialization sequence.

Original: The use of an evolutionary algorithm is motivated by its ease of implementation and natural capability to deal with continuous and discrete hyperparameters.
Revised: The use of an evolutionary algorithm is motivated by its ease of implementation and natural capability to deal with continuous and discrete hyperparameters.

Original:  P RELIMINARIES 3. B AYESIAN O PTIMIZATION
Revised:  P RELIMINARIES 3. B AYESIAN O PTIMIZATION

Original: Bayesian optimization is an optimization method for computationally intensive black box functions that consists of two main components, the surrogate model and the acquisition function.
Revised: Bayesian optimization is an optimization method for computationally intensive black box functions that consists of two main components, the surrogate model and the acquisition function.

Original: The surrogate model is a probabilistic model with mean µ and variance σ 2 , which tries to approximate the unknown black box function f , in the following also response function .
Revised: The surrogate model is a probabilistic model with mean µ and variance σ 2 , which tries to approximate the unknown black box function f , in the following also response function .

Original: The acquisition function can provide a score for each feasible solution based on the prediction of the surrogate model.
Revised: The acquisition function can provide a score for each feasible solution based on the prediction of the surrogate model.

Original: This score balances between exploration and exploitation.
Revised: This score balances between exploration and exploitation.

Original: The following steps are carried out sequentially.
Revised: The following steps are carried out sequentially.

Original: First, the feasible solution that maximizes the acquisition function is evaluated.
Revised: First, the feasible solution that maximizes the acquisition function is evaluated.

Original: In this way a new observation ( x , f ( x )) is obtained.
Revised: In this way a new observation ( x , f ( x )) is obtained.

Original: Then, the surrogate model is updated based on the entirety of all observations D .
Revised: Then, the surrogate model is updated based on the entirety of all observations D .

Original: This sequence of steps is carried out until a previously deﬁned convergence criterion occurs (e.g. exhaustion of the time budget).
Revised: This sequence of steps is carried out until a previously deﬁned convergence criterion occurs (e.g. exhaustion of the time budget).

Original: In Figure 1 we provide an example how Bayesian optimization is used to maximize a sine wave.
Revised: In Figure 1 we provide an example how Bayesian optimization is used to maximize a sine wave.

Original: Expected Improvement (Jones et al., 1998) is one of the most commonly used acquisition functions and will also be used in all our experiments.
Revised: Expected Improvement (Jones et al., 1998) is one of the most commonly used acquisition functions and will also be used in all our experiments.

Original: It is deﬁned as
Revised: It is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where y max is the largest observed value of f .
Revised: where y max is the largest observed value of f .

Original: 3. G AUSSIAN P ROCESSES
Revised: 3. G AUSSIAN P ROCESSES

Original: We have a training set D of n observations, D = { ( x i , y i ) | i = 1 , . . . , n } , where y i are noisy obser- vations of the function values
Revised: We have a training set D of n observations, D = { ( x i , y i ) | i = 1 , . . . , n } , where y i are noisy obser- vations of the function values

Original: y i
Revised: y i

Original: [Equation]
Revised: =

Original:  x i ) [Equation] ε .
Revised: f ( x i ) + ε .

Original: We assume that the noise ε is additive independent identically distributed Gaussian with variance σ 2 n .
Revised: We assume that the noise ε is additive independent identically distributed Gaussian with variance σ 2 n .

Original: For Gaussian processes (Rasmussen & Williams, 2006) we consider y i to be a random variable and the joint distribution of all y i is assumed to be multivariate Gaussian distributed:
Revised: For Gaussian processes (Rasmussen & Williams, 2006) we consider y i to be a random variable and the joint distribution of all y i is assumed to be multivariate Gaussian distributed:

Original: [Equation]
Revised: [Equation]

Original: A Gaussian process is completely speciﬁed by its mean function m , its covariance function k , and may depend on parameters θ .
Revised: A Gaussian process is completely speciﬁed by its mean function m , its covariance function k , and may depend on parameters θ .

Original: A common choice is to set m ( x i )
Revised: A common choice is to set m ( x i )

Original: = 0 .
Revised: = 0 .

Original: At inference time for instances x ∗ , the assumption is that their ground truth f ∗ is jointly Gaussian with y
Revised: At inference time for instances x ∗ , the assumption is that their ground truth f ∗ is jointly Gaussian with y

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: for brevity.
Revised: for brevity.

Original: Then, the posterior predictive distribution has mean and covariance
Revised: Then, the posterior predictive distribution has mean and covariance

Original: [Equation]
Revised: [Equation]

Original: Examples for covariance functions are the linear or squared exponential kernel.
Revised: Examples for covariance functions are the linear or squared exponential kernel.

Original: However, these kernel functions are designed by hand.
Revised: However, these kernel functions are designed by hand.

Original: The idea of deep kernel learning (Wilson et al., 2016) is to learn the kernel function.
Revised: The idea of deep kernel learning (Wilson et al., 2016) is to learn the kernel function.

Original: They propose to use a neural network ϕ to transform the input x to a latent representation which serves as input for the kernel function.
Revised: They propose to use a neural network ϕ to transform the input x to a latent representation which serves as input for the kernel function.

Original: [Equation]
Revised: [Equation]

Original:  F EW -S HOT B AYESIAN O PTIMIZATION
Revised:  F EW -S HOT B AYESIAN O PTIMIZATION

Original: Hyperparameter optimization is traditionally either tackled as an optimization problem without prior knowledge about the response function or alternatively as a multi-task or transfer learning problem.
Revised: Hyperparameter optimization is traditionally either tackled as an optimization problem without prior knowledge about the response function or alternatively as a multi-task or transfer learning problem.

Original: In the former, every search basically starts from scratch.
Revised: In the former, every search basically starts from scratch.

Original: In the latter, one or multiple models are trained that attempt to reuse knowledge of the source tasks for the target task.
Revised: In the latter, one or multiple models are trained that attempt to reuse knowledge of the source tasks for the target task.

Original: In this work we will address the problem as a few-shot problem.
Revised: In this work we will address the problem as a few-shot problem.

Original: Given T related source tasks and very few examples of the target task, we want to make reliable predictions.
Revised: Given T related source tasks and very few examples of the target task, we want to make reliable predictions.

Original: For each of the source tasks we have observations D ( t ) =
Revised: For each of the source tasks we have observations D ( t ) =

Original: { ( x ( t ) i , y ( t ) i ) } i =1 ...n ( t ) , where x ( t ) i is a hyperparameter setting and y ( t ) i is the noisy observations of f ( t ) ( x ( t ) i ) , i.e. a validation score of a machine learning model on data set t .
Revised: { ( x ( t ) i , y ( t ) i ) } i =1 ...n ( t ) , where x ( t ) i is a hyperparameter setting and y ( t ) i is the noisy observations of f ( t ) ( x ( t ) i ) , i.e. a validation score of a machine learning model on data set t .

Original: In the following we will denote the set of all data points by D :=
Revised: In the following we will denote the set of all data points by D :=

Original: (cid:83)
Revised: (cid:83)

Original: Tt =1 D ( t ) .
Revised: Tt =1 D ( t ) .

Original: Modelagnostic meta-learning (Finn et al., 2017) has become a popular choice for few-shot learning and we propose to use an adaptation for Gaussian processes (Patacchiola et al., 2020) as a surrogate model within the Bayesian optimization framework.
Revised: Modelagnostic meta-learning (Finn et al., 2017) has become a popular choice for few-shot learning and we propose to use an adaptation for Gaussian processes (Patacchiola et al., 2020) as a surrogate model within the Bayesian optimization framework.

Original: The idea is simple.
Revised: The idea is simple.

Original: A deep kernel ϕ is used to learn parameters across tasks such that all its parameters θ and w are task-independent.
Revised: A deep kernel ϕ is used to learn parameters across tasks such that all its parameters θ and w are task-independent.

Original: All task-dependent parameters are kept separate which allows to marginalize its corresponding variable [Equation] [Equation] [Equation]
Revised: All task-dependent parameters are kept separate which allows to marginalize its corresponding variable out when solely optimizing for the task-independent parameters.

Original: [Equation]

Original: out when solely optimizing for the task-independent parameters.

Original: If we assume that the posteriors over θ and w are dominated by their respective maximum likelihood estimates ˆ θ and ˆ w , we can approximate the posterior predictive distribution by
Revised: If we assume that the posteriors over θ and w are dominated by their respective maximum likelihood estimates ˆ θ and ˆ w , we can approximate the posterior predictive distribution by

Original: [Equation]
Revised: [Equation]

Original: We estimate ˆ θ and ˆ w by maximizing the log marginal likelihood for all tasks.log
Revised: We estimate ˆ θ and ˆ w by maximizing the log marginal likelihood for all tasks.

Original: [Equation]
Revised: [Equation]

Original: We maximize the marginal likelihood with stochastic gradient ascent (SGA).
Revised: We maximize the marginal likelihood with stochastic gradient ascent (SGA).

Original: Each batch contains data for one task.
Revised: Each batch contains data for one task.

Original: It is important to note that this batch data does not correspond to the data in
Revised: It is important to note that this batch data does not correspond to the data in

Original: D ( t ) , but rather a subset of it.
Revised: D ( t ) , but rather a subset of it.

Original: With small batch sizes, this is an efﬁcient way to train the Gaussian process, regardless of how many tasks knowledge is transferred from or how large D ( t ) is.
Revised: With small batch sizes, this is an efﬁcient way to train the Gaussian process, regardless of how many tasks knowledge is transferred from or how large D ( t ) is.

Original: It is important to note that there is a correlation between all data per task and not just one batch.
Revised: It is important to note that there is a correlation between all data per task and not just one batch.

Original: Hence the stochastic gradient is a biased estimator of the full gradient.
Revised: Hence the stochastic gradient is a biased estimator of the full gradient.

Original: For a long time we lacked the theoretical understanding of how SGA would behave in this situation.
Revised: For a long time we lacked the theoretical understanding of how SGA would behave in this situation.

Original: Fortunately, recent work (Chen et al., 2020) proved that SGA still successfully converges and restores model parameters in these cases.
Revised: Fortunately, recent work (Chen et al., 2020) proved that SGA still successfully converges and restores model parameters in these cases.

Original: The results by Chen et al.
Revised: The results by Chen et al.

Original: (2020) guarantee a O (1 /K ) optimization margin of error, where K is the number of iterations.
Revised: (2020) guarantee a O (1 /K ) optimization margin of error, where K is the number of iterations.

Original: The training works as follows.
Revised: The training works as follows.

Original: In each iteration, a task t is sampled uniformly at random from all
Revised: In each iteration, a task t is sampled uniformly at random from all

Original: T tasks.
Revised: T tasks.

Original: Then we sample a batch of training instances uniformly at random from D ( t ) .
Revised: Then we sample a batch of training instances uniformly at random from D ( t ) .

Original: Finally, we calculate the log marginal likelihood for this batch (Equation 8) and update the kernel parameters with one step in the direction of the gradient.
Revised: Finally, we calculate the log marginal likelihood for this batch (Equation 8) and update the kernel parameters with one step in the direction of the gradient.

Original: In hyperparameter optimization, we are only interested in the hyperparameter setting that works best according to a predeﬁned metric of interest.
Revised: In hyperparameter optimization, we are only interested in the hyperparameter setting that works best according to a predeﬁned metric of interest.

Original: Therefore, only the ranking of the hyperparameter settings is important, not the actual value of the metric.
Revised: Therefore, only the ranking of the hyperparameter settings is important, not the actual value of the metric.

Original: Therefore, we are interested in a surrogate model whose prediction strongly correlate with the response function and the squared error is of little to no interest for us.
Revised: Therefore, we are interested in a surrogate model whose prediction strongly correlate with the response function and the squared error is of little to no interest for us.

Original: In practice, however, the minimum / maximum score and the range of values differ signiﬁcantly between the tasks which makes it challenging to obtain a strongly correlating surrogate model.
Revised: In practice, however, the minimum / maximum score and the range of values differ signiﬁcantly between the tasks which makes it challenging to obtain a strongly correlating surrogate model.

Original: The most common way to address this problem is to normalize the labels, e.g. by z-normalizing or scaling the labels to [0 , 1] per data set.
Revised: The most common way to address this problem is to normalize the labels, e.g. by z-normalizing or scaling the labels to [0 , 1] per data set.

Original: However, this does not fully solve the problem.
Revised: However, this does not fully solve the problem.

Original: The main problem is that this label normalization must also be applied to the target task.
Revised: The main problem is that this label normalization must also be applied to the target task.

Original: This is not possible with a satisfactory degree of accuracy, especially when only a few examples are available, since the approximate values for minimum / maximum or mean / standard deviation are insufﬁciently accurate.[Figure]
Revised: This is not possible with a satisfactory degree of accuracy, especially when only a few examples are available, since the approximate values for minimum / maximum or mean / standard deviation are insufﬁciently accurate.

Original: Since our proposed method scales easily to any number of tasks, we can afford to consider another option.
Revised: Since our proposed method scales easily to any number of tasks, we can afford to consider another option.

Original: We propose a task augmentation strategy that addresses the label normalization issue by randomly scaling the labels for each batch.
Revised: We propose a task augmentation strategy that addresses the label normalization issue by randomly scaling the labels for each batch.

Original: Since y min and y max are the minimum and maximum values for all T tasks, we can generate augmented tasks for each batch B = { x i , y i } i
Revised: Since y min and y max are the minimum and maximum values for all T tasks, we can generate augmented tasks for each batch B = { x i , y i } i

Original: =1 ,...,b ∼ D ( t ) , where b is the batch size, as follows.
Revised: =1 ,...,b ∼ D ( t ) , where b is the batch size, as follows.

Original: A lower and upper limit is sampled for each sample batch,
Revised: A lower and upper limit is sampled for each sample batch,

Original: [Equation]
Revised: [Equation]

Original:  such that l < u holds.
Revised: (10) such that l < u holds.

Original: Then the labels for that batch are scaled to this range,
Revised: Then the labels for that batch are scaled to this range,

Original: [Equation]
Revised: [Equation]

Original: No further changes are required.
Revised: No further changes are required.

Original: We summarize this procedure in Algorithm 1.
Revised: We summarize this procedure in Algorithm 1.

Original: The idea here is to learn a representation that is invariant with respect to various offsets and ranges so that the target task does not require normalization.
Revised: The idea here is to learn a representation that is invariant with respect to various offsets and ranges so that the target task does not require normalization.

Original: This strategy is not possible for other hyperparameter optimization methods, since this either increases the training data set size even further and thus becomes  computationally impossible (Bardenet et al., 2013; Swersky et al., 2013; Yogatama & Mann, 2014) or thousands of new tasks would have to be generated, which is also is not practical (Springenberg et al., 2016;Perrone et al., 2018;
Revised: This strategy is not possible for other hyperparameter optimization methods, since this either increases the training data set size even further and thus becomes while not done do − [Figure] computationally impossible (Bardenet et al., 2013; Swersky et al., 2013; Yogatama & Mann, 2014) or thousands of new tasks would have to be generated, which is also is not practical (Springenberg et al., 2016;

Original:  Wistuba et al., 2018; Feurer et al., 2018).
Revised: Perrone et al., 2018; Wistuba et al., 2018; Feurer et al., 2018).

Original: At test time, the posterior predictive distribution for the target task T + 1 is computed according to Equation 5.
Revised: At test time, the posterior predictive distribution or the target task T + 1 is computed according to Equation 5.

Original: In practice, the target data set can be very different to the learned prior.
Revised: In practice, the target data set can be very different to the learned prior.

Original: For this reason, the deep kernel parameters are ﬁne-tuned on
Revised: For this reason, the deep kernel parameters are ﬁne-tuned on

Original: the target task’s data for few epochs.
Revised: the target task’s data for few epochs.

Original: Our task augmentation strategy described earlier has resulted in a model that has become invariant for different scales of y .
Revised: Our task augmentation strategy described earlier has resulted in a model that has become invariant for different scales of y .

Original: For this reason we do not apply this strategy to the target task T + 

Revised: For this reason we do not apply this strategy to the target task T +

Original: and only use it for all source tasks 1 to T .
Revised: and only use it for all source tasks 1 to T .

Original: We discuss the usefulness of this step in more detail when we discuss the empirical results.
Revised: We discuss the usefulness of this step in more detail when we discuss the empirical results.

Original:  A M OTIVATING E XAMPLE
Revised:  A M OTIVATING E XAMPLE

Original: We would like to ﬁrst motivate our method, FSBO, with an example, similar to the one described in
Revised: We would like to ﬁrst motivate our method, FSBO, with an example, similar to the one described in

Original: Nichol et al.
Revised: Nichol et al.

Original: The aim is to ﬁnd the argument x that maximizes a one-dimensional sine wave of the form f ( t ) ( x ) = a ( t ) sin( x + b ( t ) ) .
Revised: The aim is to ﬁnd the argument x that maximizes a one-dimensional sine wave of the form f ( t ) ( x ) = a ( t ) sin( x + b ( t ) ) .

Original: It is not known that the function is a sine wave and that one only has to determine the amplitude a and phase b .
Revised: It is not known that the function is a sine wave and that one only has to determine the amplitude a and phase b .

Original: However, information about other sine waves  is available to the optimizer which are considered to be related.
Revised: However, information about other sine waves [Figure] is available to the optimizer which are considered to be related.

Original: These source tasks t = 1 , . .
Revised: These source tasks t = 1 , . .

Original: , T are generated randomly with a ∼ U (0 . 1 , 5) and b ∼ U (0 , 2 π ) .
Revised: , T are generated randomly with a ∼ U (0 . 1 , 5) and b ∼ U (0 , 2 π ) .

Original: 50 examples ( x ( t ) i , f ( t ) ( x ( t ) i )) are available for each task t , whereby the points x ( t ) i ∈ [ − 5 , 5] are evenly spaced.
Revised: 50 examples ( x ( t ) i , f ( t ) ( x ( t ) i )) are available for each task t , whereby the points x ( t ) i ∈ [ − 5 , 5] are evenly spaced.

Original: It should be noted at this point that the expected value for each x i is 0.
Revised: It should be noted at this point that the expected value for each x i is 0.

Original: Thus, training on the data described above leads to a model predicting 0 for every x i .
Revised: Thus, training on the data described above leads to a model predicting 0 for every x i .

Original: However, the used meta-learning procedure allows for accurately reconstructing the underlying sine wave after only a few examples (see Figure 1).
Revised: However, the used meta-learning procedure allows for accurately reconstructing the underlying sine wave after only a few examples (see Figure 1).

Original: Provided that the response function of the original task is similar to the target task, this is a very promising technique for accelerating the search for good hyperparameter settings.
Revised: Provided that the response function of the original task is similar to the target task, this is a very promising technique for accelerating the search for good hyperparameter settings.

Original:  A D ATA -D RIVEN I NITIALIZATION
Revised:  A D ATA -D RIVEN I NITIALIZATION

Original: The proposed few-shot surrogate model requires only a few examples before reliable predictions can be made for the target data set.
Revised: The proposed few-shot surrogate model requires only a few examples before reliable predictions can be made for the target data set.

Original: How do you choose the initial hyper parameter settings?
Revised: How do you choose the initial hyper parameter settings?

Original: The simplest idea would be to pick them at random or use Latin Hypercube Sampling (LHS) (McKay et al., 1979).
Revised: The simplest idea would be to pick them at random or use Latin Hypercube Sampling (LHS) (McKay et al., 1979).

Original: Since we already have data from various tasks and a surrogate model that can impute missing values, we propose a data-driven warm start approach.
Revised: Since we already have data from various tasks and a surrogate model that can impute missing values, we propose a data-driven warm start approach.

Original: If we evaluate the performance of a hyperparameter optimization algorithm with a loss function L , we use an evolutionary algorithm to ﬁnd a set with I hyperparameter settings which minimizes this loss on the source tasks.
Revised: If we evaluate the performance of a hyperparameter optimization algorithm with a loss function L , we use an evolutionary algorithm to ﬁnd a set with I hyperparameter settings which minimizes this loss on the source tasks.

Original: [Equation]
Revised: [Equation]

Original: The loss of a set of hyperparameters depends on the response function values for each of the elements.
Revised: The loss of a set of hyperparameters depends on the response function values for each of the elements.

Original: Since these are not available for every x , we approximate f ( t ) ( x ) with the prediction of the surrogate model described in the last section whenever this is necessary.
Revised: Since these are not available for every x , we approximate f ( t ) ( x ) with the prediction of the surrogate model described in the last section whenever this is necessary.

Original: Arbitrary loss function can be considered.
Revised: Arbitrary loss function can be considered.

Original: The speciﬁc loss function used in our experiments is the normalized regret and is deﬁned on the right side of Equation 12, where ˜ f ( t ) is derived from f ( t ) by scaling it to [0 , 1] range:
Revised: The speciﬁc loss function used in our experiments is the normalized regret and is deﬁned on the right side of Equation 12, where ˜ f ( t ) is derived from f ( t ) by scaling it to [0 , 1] range:

Original: [Equation]
Revised: [Equation]

Original: where f ( t ) max and f ( t ) min are the maximum and minimum of f ( t ) , respectively.
Revised: where f ( t ) max and f ( t ) min are the maximum and minimum of f ( t ) , respectively.

Original: The evolutionary algorithm works as follows.
Revised: The evolutionary algorithm works as follows.

Original: We initialize the population with sets containing I random settings, the settings being sampled in proportion to their performance according to the predictions.
Revised: We initialize the population with sets containing I random settings, the settings being sampled in proportion to their performance according to the predictions.

Original: Precisely, a setting x is sampled proportional to
Revised: Precisely, a setting x is sampled proportional to

Original: [Equation]
Revised: [Equation]

Original: The best sets are then selected to be reﬁned using an evolutionary algorithm.
Revised: The best sets are then selected to be reﬁned using an evolutionary algorithm.

Original: The algorithm randomly chooses either to mutate a set or to perform a crossover operation between two sets.
Revised: The algorithm randomly chooses either to mutate a set or to perform a crossover operation between two sets.

Original: When mutating a set, a setting is removed uniformly at random and a new setting is added proportional to its predicted performance (Figure 2, left).
Revised: When mutating a set, a setting is removed uniformly at random and a new setting is added proportional to its predicted performance (Figure 2, left).

Original: The crossover operation creates a new set and adds elements from the union of the parent sets until the new set has I settings (Figure 2, right).
Revised: The crossover operation creates a new set and adds elements from the union of the parent sets until the new set has I settings (Figure 2, right).

Original: The new set is added to the population.
Revised: The new set is added to the population.

Original: After 100,000 steps, the algorithm is stopped and the best set is used as the set of initial hyperparameter settings.
Revised: After 100,000 steps, the algorithm is stopped and the best set is used as the set of initial hyperparameter settings.

Original: In Figure 4 we compare this warm start initialization with the simple random or LHS initialization.
Revised: In Figure 4 we compare this warm start initialization with the simple random or LHS initialization.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We used three different optimization problems to compare the different hyperparameter optimization methods: AdaBoost, GLMNet, and SVM.
Revised: We used three different optimization problems to compare the different hyperparameter optimization methods: AdaBoost, GLMNet, and SVM.

Original: We created the GLMNet and SVM metadata set by downloading the 30 data sets with the most reported hyperparameter settings from OpenML for each problem.
Revised: We created the GLMNet and SVM metadata set by downloading the 30 data sets with the most reported hyperparameter settings from OpenML for each problem.

Original: The AdaBoost data set is publicly available (Wistuba et al., 2018).
Revised: The AdaBoost data set is publicly available (Wistuba et al., 2018).

Original: The number of settings per data set varies, the number of settings across all tasks for the GLMNet problem is approximately 800,000.
Revised: The number of settings per data set varies, the number of settings across all tasks for the GLMNet problem is approximately 800,000.

Original: See the appendix for more details about the metadata sets.
Revised: See the appendix for more details about the metadata sets.

Original: We compare the following list of hyperparameter optimization methods.
Revised: We compare the following list of hyperparameter optimization methods.

Original: [Figure]

Original: Random Search This is a simple but strong baseline for hyperparameter optimization (Bergstra &
Revised: Random Search This is a simple but strong baseline for hyperparameter optimization (Bergstra &

Original: Bengio, 2012).
Revised: Bengio, 2012).

Original: Hyperparameters settings are selected uniformly at random from the search space.
Revised: Hyperparameters settings are selected uniformly at random from the search space.

Original: Gaussian Process (GP) Bayesian optimization with a Gaussian process as a surrogate model is a standard and strong baseline (Snoek et al., 2012).
Revised: Gaussian Process (GP) Bayesian optimization with a Gaussian process as a surrogate model is a standard and strong baseline (Snoek et al., 2012).

Original: We use a Mat´ern 5/2 kernel with ARD and rely on the scikit-optimize implementation.
Revised: We use a Mat ´ ern 5/2 kernel with ARD and rely on the scikit-optimize implementation.

Original: We compare with two variations.
Revised: We compare with two variations.

Original: The ﬁrst is the vanilla method, which uses Latin Hypercube Sampling (LHS) with design size of 10 to initialize.
Revised: The ﬁrst is the vanilla method, which uses Latin Hypercube Sampling (LHS) with design size of 10 to initialize.

Original: The second method uses the warm start (WS) method described in Section 6 to ﬁnd an initial set ofsettings to use the knowledge from other data sets.
Revised: The second method uses the warm start (WS) method described in Section 6 to ﬁnd an initial set ofsettings to use the knowledge from other data sets.

Original: RGPE RGPE (Feurer et al., 2018) is one of the latest examples of methods that use GP ensembles to transfer knowledge across task (Wistuba et al., 2016;
Revised: RGPE RGPE (Feurer et al., 2018) is one of the latest examples of methods that use GP ensembles to transfer knowledge across task (Wistuba et al., 2016;

Original: Schilling et al., 2016; Wistuba et al., 2018).
Revised: Schilling et al., 2016; Wistuba et al., 2018).

Original: In this case a GP is trained for each individual task and in a ﬁnal step the predictions of the GPs are aggregated.
Revised: In this case a GP is trained for each individual task and in a ﬁnal step the predictions of the GPs are aggregated.

Original: These ensembles scale linearly in the number of tasks, but the computation time is still cubic and the space requirement is still quadratic in the number of data points per task.
Revised: These ensembles scale linearly in the number of tasks, but the computation time is still cubic and the space requirement is still quadratic in the number of data points per task.

Original: For this reason we can only present results for AdaBoost and not for the other optimization problems, which have signiﬁcantly more data points.
Revised: For this reason we can only present results for AdaBoost and not for the other optimization problems, which have signiﬁcantly more data points.

Original: MetaBO Using the acquisition function instead of the surrogate model for transfer learning is another option.
Revised: MetaBO Using the acquisition function instead of the surrogate model for transfer learning is another option.

Original: MetaBO is the state-of-the-art transfer acquisition function and was presented last year at ICLR (Volpp et al., 2020).
Revised: MetaBO is the state-of-the-art transfer acquisition function and was presented last year at ICLR (Volpp et al., 2020).

Original: We use the implementation provided by the authors.
Revised: We use the implementation provided by the authors.

Original: ABLR
Revised: ABLR

Original: The state-of-the-art surrogate model for scalable hyperparameter transfer learning (Perrone et al., 2018).
Revised: The state-of-the-art surrogate model for scalable hyperparameter transfer learning (Perrone et al., 2018).

Original: This method uses a multi-task GP which combines a linear kernel with a neural network and scales to very large data sets.
Revised: This method uses a multi-task GP which combines a linear kernel with a neural network and scales to very large data sets.

Original: Transfer learning is achieved by sharing the neural network parameters across different tasks.
Revised: Transfer learning is achieved by sharing the neural network parameters across different tasks.

Original: By ABLR (WS) we are referring to a version of ABLR that uses the same warm start as FSBO.
Revised: By ABLR (WS) we are referring to a version of ABLR that uses the same warm start as FSBO.

Original: Multi-Head GPs This closely resembles ABLR but uses the same deep kernel as our proposed method.
Revised: Multi-Head GPs This closely resembles ABLR but uses the same deep kernel as our proposed method.

Original: The main difference from our proposed method is that it is using a GP for every task, only shares the neural network across tasks, and uses standard stochastic gradient ascent.
Revised: The main difference from our proposed method is that it is using a GP for every task, only shares the neural network across tasks, and uses standard stochastic gradient ascent.

Original: Few-Shot Bayesian Optimization (FSBO)
Revised: Few-Shot Bayesian Optimization (FSBO)

Original: Our proposed method described in Section 4.
Revised: Our proposed method described in Section 4.

Original: The deep kernel is composed of a two-layer neural network ( 128 → 128 ) with ReLU activations and a squared-exponential kernel.
Revised: The deep kernel is composed of a two-layer neural network ( 128 → 128 ) with ReLU activations and a squared-exponential kernel.

Original: We use the Adam optimizer with learning rate 10 − 3 and a batch size of ﬁfty.
Revised: We use the Adam optimizer with learning rate 10 − 3 and a batch size of ﬁfty.

Original: The warm start length is ﬁve.
Revised: The warm start length is ﬁve.

Original: The experiments are repeated ten times and evaluated in a leave-one-task-out cross-validation.
Revised: The experiments are repeated ten times and evaluated in a leave-one-task-out cross-validation.

Original: This means that all transfer learning methods use one task as the target task and all other tasks as source tasks.
Revised: This means that all transfer learning methods use one task as the target task and all other tasks as source tasks.

Original: For AdaBoost we use the same train/test split as used by Volpp et al.
Revised: For AdaBoost we use the same train/test split as used by Volpp et al.

Original: (2020) instead.
Revised: (2020) instead.

Original: We report the aggregated results for all tasks within one problem class with respect to the mean of normalized regrets.
Revised: We report the aggregated results for all tasks within one problem class with respect to the mean of normalized regrets.

Original: The normalized regret for a task is obtained by ﬁrst scaling the response function values between 0 and 1 before calculating the regret (Wistuba et al., 2018), i.e.
Revised: The normalized regret for a task is obtained by ﬁrst scaling the response function values between 0 and 1 before calculating the regret (Wistuba et al., 2018), i.e.

Original: [Equation]
Revised: [Equation]

Original: (15) where ˜ f ( t ) is the normalized response function (Equation 13),
Revised: [Table] where ˜ f ( t ) is the normalized response function (Equation 13),

Original: ˜ f ( t ) min is the global minimum for task t, and x min is the best discovered hyperparameter setting by the optimizer.
Revised: ˜ f ( t ) min is the global minimum for task t

Original: [Table]
Revised: [Table]

Revised: and x min is the best discovered hyperparameter setting by the optimizer.

Original:  H YPERPARAMETER O PTIMIZATION
Revised: 7. H YPERPARAMETER O PTIMIZATION

Original: We conduct experiments on three different metadata sets and report the aggregated mean of normalized regrets in Table 1.
Revised: We conduct experiments on three different metadata sets and report the aggregated mean of normalized regrets in Table 1.

Original: The best results are in bold.
Revised: The best results are in bold.

Original: Results that are not signiﬁcantly worse than the best are in italics.
Revised: Results that are not signiﬁcantly worse than the best are in italics.

Original: We determined the signiﬁcance using the Wilcoxon signed rank test with a conﬁdence level of 95%.
Revised: We determined the signiﬁcance using the Wilcoxon signed rank test with a conﬁdence level of 95%.

Original: Results are reported after every 33 trials of Bayesian optimization for GLMNet and SVM.
Revised: Results are reported after every 33 trials of Bayesian optimization for GLMNet and SVM.

Original: Since AdaBoost has fewer test examples, we report its results in shorter intervals.
Revised: Since AdaBoost has fewer test examples, we report its results in shorter intervals.

Original: Our proposed FSBO method outperforms all other transfer methods in all three tasks.
Revised: Our proposed FSBO method outperforms all other transfer methods in all three tasks.

Original: Results are signiﬁcantly better but in the case of AdaBoost where GP (WS) achieves similar but on average worse results.
Revised: Results are signiﬁcantly better but in the case of AdaBoost where GP (WS) achieves similar but on average worse results.

Original: The results obtained for MetaBO are the worst.
Revised: The results obtained for MetaBO are the worst.

Original: We contacted Volpp et al.
Revised: We contacted Volpp et al.

Original: and they conﬁrmed that our results are obtained correctly.
Revised: and they conﬁrmed that our results are obtained correctly.

Original: The problem is apparently that the Reinforcement Learning method does not work well for larger number of trials.
Revised: The problem is apparently that the Reinforcement Learning method does not work well for larger number of trials.

Original: We provide a longer discussion in the appendix.
Revised: We provide a longer discussion in the appendix.

Original: Also ABLR and the very related Multi-Head GP are not performing very well.
Revised: Also ABLR and the very related Multi-Head GP are not performing very well.

Original: As we show in the next section, one possible reason for this might be because the neural network parameters are ﬁxed which will prohibit a possibly required adaptation to the target task.
Revised: As we show in the next section, one possible reason for this might be because the neural network parameters are ﬁxed which will prohibit a possibly required adaptation to the target task.

Original: The vanilla GP and its transfer variant that uses a warm start turn out to be among the strongest baselines.
Revised: The vanilla GP and its transfer variant that uses a warm start turn out to be among the strongest baselines.

Original: This is in particular true for the warm start version which is the second best method.
Revised: This is in particular true for the warm start version which is the second best method.

Original: This simple baseline of knowledge transfer is often not taken into account when comparing transfer surrogates, although it is easy to implement.
Revised: This simple baseline of knowledge transfer is often not taken into account when comparing transfer surrogates, although it is easy to implement.

Original: Due to the very competitive results, we recommend using it as a standard baseline to assess the usefulness of new transfer surrogates.
Revised: Due to the very competitive results, we recommend using it as a standard baseline to assess the usefulness of new transfer surrogates.

Original: 7. C OMPONENT C ONTRIBUTIONS
Revised: 7. C OMPONENT C ONTRIBUTIONS

Original: In this section we highlight the various components that make a signiﬁcant difference to ABLR.
Revised: In this section we highlight the various components that make a signiﬁcant difference to ABLR.

Original: The results for each setup are reported in Figure 3.
Revised: The results for each setup are reported in Figure 3.

Original: The Multi-Head GP is a surrogate model that shares the neural network between tasks but uses a separate Gaussian process for each task.
Revised: The Multi-Head GP is a surrogate model that shares the neural network between tasks but uses a separate Gaussian process for each task.

Original: It closely resembles the idea of ABLR but is closer to our proposed implementation of FSBO.
Revised: It closely resembles the idea of ABLR but is closer to our proposed implementation of FSBO.

Original: Starting from this conﬁguration, we add various components that will eventually lead to our proposed model
Revised: Starting from this conﬁguration, we add various components that will eventually lead to our proposed model

Original: FSBO.
Revised: FSBO.

Original: We consider Multi-Head GP (WS), a version that additionally uses the warm start method described in Section 6 instead of a random initialization.
Revised: We consider Multi-Head GP (WS), a version that additionally uses the warm start method described in Section 6 instead of a random initialization.

Original: Multi-Head GP (ﬁne-tune)
Revised: Multi-Head GP (ﬁne-tune)

Original: not only updates the kernel parameters when a new observation is received but also ﬁne-tunes the parameters of the neural network.
Revised: not only updates the kernel parameters when a new observation is received but also ﬁne-tunes the parameters of the neural network.

Original: Finally, FSBO is our proposed method, which uses only one Gaussian process for all tasks.
Revised: Finally, FSBO is our proposed method, which uses only one Gaussian process for all tasks.

Original: We see that all Multi-Head GP versions have a problem adapting to the target tasks efﬁciently.
Revised: We see that all Multi-Head GP versions have a problem adapting to the target tasks efﬁciently.

Original: Fine-tuning the deep kernel is an important part of learning FSBO.
Revised: Fine-tuning the deep kernel is an important part of learning FSBO.

Original: Although FSBO outperforms all Multi-Head GP versions without ﬁne-tuning, the additional use of it makes for a signiﬁcant improvement.
Revised: Although FSBO outperforms all Multi-Head GP versions without ﬁne-tuning, the additional use of it makes for a signiﬁcant improvement.

Original: We analyzed the reason for this in more detail.
Revised: We analyzed the reason for this in more detail.

Original: We observed that the learned neural  network provides a strong prior that leads to strong exploitation behavior.
Revised: We observed that the learned neural 1 Out-of-memory exception due to too large data. [Figure] [Figure] network provides a strong prior that leads to strong exploitation behavior.

Original: Fine-tuning prevents this and thus ensures that the method does not get stuck in local optima.
Revised: Fine-tuning prevents this and thus ensures that the method does not get stuck in local optima.

Original: [Figure] [Figure] [Figure]
Revised: 7. D ATA -D RIVEN W ARM S TART

Original: The use of a warm start is not the main contribution of this paper but it is interesting to explore further nonetheless.
Revised: The use of a warm start is not the main contribution of this paper but it is interesting to explore further nonetheless.

Original: First of all, for some methods this is the only differentiating factor.
Revised: First of all, for some methods this is the only differentiating factor.

Original: We compare our suggested warm start initialization with a random and a Latin hypercube sampling (LHS) initialization in Figure 4.
Revised: We compare our suggested warm start initialization with a random and a Latin hypercube sampling (LHS) initialization in Figure 4.

Original: Considering that GP (WS) always performed better than GP (LHS) in Table 1, one would expect that the warm start itself also performs better than LHS.
Revised: Considering that GP (WS) always performed better than GP (LHS) in Table 1, one would expect that the warm start itself also performs better than LHS.

Original: While this is the case for
Revised: While this is the case for

Original: GLMNet and SVM, there are no signiﬁcant differences for the AdaBoost problem.
Revised: GLMNet and SVM, there are no signiﬁcant differences for the AdaBoost problem.

Original: Our assumption is that in this case the warm start might not have always found good settings as part of the initialization, it still explored areas in the search space close to the optimum.
Revised: Our assumption is that in this case the warm start might not have always found good settings as part of the initialization, it still explored areas in the search space close to the optimum.

Original: This would facilitate ﬁnding a better solution in one of the subsequent steps of Bayesian optimization.
Revised: This would facilitate ﬁnding a better solution in one of the subsequent steps of Bayesian optimization.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: In this work, we propose to rethink hyperparameter optimization as a few-shot learning problem in which we train a shared deep surrogate model to quickly adapt (with few response evaluations) to the response function of a new task.
Revised: In this work, we propose to rethink hyperparameter optimization as a few-shot learning problem in which we train a shared deep surrogate model to quickly adapt (with few response evaluations) to the response function of a new task.

Original: We propose the use of a deep kernel network for a Gaussian process surrogate that is meta-learned in an end-to-end fashion in order to jointly approximate the response functions of a collection of training data sets.
Revised: We propose the use of a deep kernel network for a Gaussian process surrogate that is meta-learned in an end-to-end fashion in order to jointly approximate the response functions of a collection of training data sets.

Original: This few-shot surrogate model is used for two different purposes.
Revised: This few-shot surrogate model is used for two different purposes.

Original: First, we use it in combination with an evolutionary algorithm in order to estimate a data-driven warm start initialization for Bayesian optimization.
Revised: First, we use it in combination with an evolutionary algorithm in order to estimate a data-driven warm start initialization for Bayesian optimization.

Original: Second, we use it directly for
Revised: Second, we use it directly for

Original: Bayesian optimization.
Revised: Bayesian optimization.

Original: In our empirical evaluation on three hyperparameter optimization problems, we observe signiﬁcantly better results than with state-of-the-art methods that use transfer learning.
Revised: In our empirical evaluation on three hyperparameter optimization problems, we observe signiﬁcantly better results than with state-of-the-art methods that use transfer learning.

Revised: A CKNOWLEDGMENTS

Revised: This work has been supported by European Union’s Horizon 2020 research and innovation programme under grant number 951911 - AI4Media.

Revised: Prof. Grabocka is thankful to the Eva Mayr-Stihl

Revised: Foundation for their generous research grant.

Original: A H YPERPARAMETER O PTIMIZATION
Revised: A H YPERPARAMETER O PTIMIZATION

Original: In the main paper we report the normalized regret after every 33 trials of Bayesian optimization in a table.
Revised: In the main paper we report the normalized regret after every 33 trials of Bayesian optimization in a table.

Original: This allows us to also report statistical signiﬁcance.
Revised: This allows us to also report statistical signiﬁcance.

Original: In Figure 5 we show results for alltrials.
Revised: In Figure 5 we show results for alltrials.

Original: The conclusions remain unchanged.
Revised: The conclusions remain unchanged.

Original: FSBO provides consistently the best results.
Revised: FSBO provides consistently the best results.

Original: B C HALLENGES WITH M ETA BO
Revised: B C HALLENGES WITH M ETA BO

Original: The results reported in the main paper for MetaBO seem not to align with the numbers reported by
Revised: The results reported in the main paper for MetaBO seem not to align with the numbers reported by

Original: Volpp et al.
Revised: Volpp et al.

Original: (2020), in particular for AdaBoost.
Revised: (2020), in particular for AdaBoost.

Original: In order to understand this behavior, we conducted a deeper analysis for the AdaBoost metadata set which was used in the evaluation of Volpp et al.
Revised: In order to understand this behavior, we conducted a deeper analysis for the AdaBoost metadata set which was used in the evaluation of Volpp et al.

Original: (2020) as well.
Revised: (2020) as well.

Original: Using the authors’ code 2 , we executed the same scripts used by the authors to report their results for T = 15 trials.
Revised: Using the authors’ code 2 , we executed the same scripts used by the authors to report their results for T = 15 trials.

Original: Furthermore, we changed two lines in the scripts to train MetaBO for
Revised: Furthermore, we changed two lines in the scripts to train MetaBO for

Original: T = 50 , the number of trials considered for AdaBoost in our experiment.
Revised: T = 50 , the number of trials considered for AdaBoost in our experiment.

Original: We report the outcome of these two experiments in Figure 6.
Revised: We report the outcome of these two experiments in Figure 6.

Original: In the two left plots we show how the learned policies for
Revised: In the two left plots we show how the learned policies for

Original: T = 15 and T = 50 perform on validation and test during training.
Revised: T = 15 and T = 50 perform on validation and test during training.

Original: The authors’ default setting for the total number of iterations is 2000.
Revised: The authors’ default setting for the total number of iterations is 2000.

Original: However, we noticed a sudden drop in regret for the setting T = 50 around 2000 iterations.
Revised: However, we noticed a sudden drop in regret for the setting T = 50 around 2000 iterations.

Original: For that reason we assumed that further training might help to further improve MetaBO and increased it to 5000 iterations.
Revised: For that reason we assumed that further training might help to further improve MetaBO and increased it to 5000 iterations.

Original: As can be seen in the middle plot, that was not the case.
Revised: As can be seen in the middle plot, that was not the case.

Original: We observe that the policy learned for T = 15 has a lower regret than the one for T = 50 even though it is only trying 15 compared to 50 trials.
Revised: We observe that the policy learned for T = 15 has a lower regret than the one for T = 50 even though it is only trying 15 compared to 50 trials.

Original: In the right plot we show the results compared to a simple random search baseline.
Revised: In the right plot we show the results compared to a simple random search baseline.

Original: The results for T = 15 are in line with those reported by Volpp et al.
Revised: The results for T = 15 are in line with those reported by Volpp et al.

Original: We contacted the authors and they conﬁrmed that we use their code correctly.
Revised: We contacted the authors and they conﬁrmed that we use their code correctly.

Original: Their explanation is that the increase of episode length makes this a harder problem for Reinforcement Learning.
Revised: Their explanation is that the increase of episode length makes this a harder problem for Reinforcement Learning.

Original: This results in worse results for T = 50 and even worse for
Revised: This results in worse results for T = 50 and even worse for

Original: T = 100 in case of the GLMNet and SVM problem.
Revised: T = 100 in case of the GLMNet and SVM problem.

Original: They proposed to randomly vary T betweenand 50 during the training phase.
Revised: They proposed to randomly vary T betweenand 50 during the training phase.

Original: We also considered this training protocol and report the results in
Revised: We also considered this training protocol and report the results in

Original: Figure 6.
Revised: Figure 6.

Original: This improves over training only with T = 50 but does not improve over a simple random search.
Revised: This improves over training only with T = 50 but does not improve over a simple random search.

Original: C M ETADATA
Revised: C M ETADATA

Original: We created the GLMNet and SVM metadata using OpenML.
Revised: We created the GLMNet and SVM metadata using OpenML.

Original: We chose the 30 OpenML ﬂows with the most observations.
Revised: We chose the 30 OpenML ﬂows with the most observations.

Original: We also limited ourselves to the uploaded results from user with ID(OpenML Bot R), who uploaded the majority of all runs, to ensure that the metadata was generated under similar circumstances.
Revised: We also limited ourselves to the uploaded results from user with ID(OpenML Bot R), who uploaded the majority of all runs, to ensure that the metadata was generated under similar circumstances.

Original: In some cases the choice of a single hyperparameter is not indicated.
Revised: In some cases the choice of a single hyperparameter is not indicated.

Original: In such a case, we assume that the default value was used.
Revised: In such a case, we assume that the default value was used.

Original: The GLMNet metadata has two continous hyperparameters: the elastic-net mixing parameter α ∈ [0 , 1] and the regularization parameter λ ∈ [2 − 10 , 2 10 ] .
Revised: The GLMNet metadata has two continous hyperparameters: the elastic-net mixing parameter α ∈ [0 , 1] and the regularization parameter λ ∈ [2 − 10 , 2 10 ] .

Original: The SVM metadata has two mandatory hyperparameters and two conditional
Revised: The SVM metadata has two mandatory hyperparameters and two conditional

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: hyperparameters.
Revised: hyperparameters.

Original: The kernel (linear, polynomial or RBF) and the continuous trade-off parameter
Revised: The kernel (linear, polynomial or RBF) and the continuous trade-off parameter

Original: C ∈ [2 − 10 , 2 10 ] must always be selected.
Revised: C ∈ [2 − 10 , 2 10 ] must always be selected.

Original: The degree d ∈ { 2 , 3 , 4 , 5 } is only considered for the polynomial kernel and the bandwidth γ ∈ [2 − 10 , 2 10 ] is only considered for the RBF kernel.
Revised: The degree d ∈ { 2 , 3 , 4 , 5 } is only considered for the polynomial kernel and the bandwidth γ ∈ [2 − 10 , 2 10 ] is only considered for the RBF kernel.

Original: We further use the AdaBoost metadata which has been used to evaluate MetaBO (Volpp et al., 2020).
Revised: We further use the AdaBoost metadata which has been used to evaluate MetaBO (Volpp et al., 2020).

Original: According to Wistuba et al.
Revised: According to Wistuba et al.

Original: (2015b)
Revised: (2015b)

Original: , this metadata was created using AdaBoost (K ´ egl &
Revised: , this metadata was created using AdaBoost (K ´ egl &

Original: Busa-Fekete, 2009) with decision products as weak learners.
Revised: Busa-Fekete, 2009) with decision products as weak learners.

Original: The authors designed a grid over the two continuous hyperparameters: the number of iterations and the number of product terms.
Revised: The authors designed a grid over the two continuous hyperparameters: the number of iterations and the number of product terms.

Original: The number of iterations was chosen from { 2 , 5 , 10 , 20 , 50 , 100 , 200 , 500 , 10 3 , 2 · 10 3 , 5 · 10 3 , 10 4 } , the number of product term was chosen from { 2 , 3 , 4 , 5 , 7 , 10 , 15 , 20 , 30 } .
Revised: The number of iterations was chosen from { 2 , 5 , 10 , 20 , 50 , 100 , 200 , 500 , 10 3 , 2 · 10 3 , 5 · 10 3 , 10 4 } , the number of product term was chosen from { 2 , 3 , 4 , 5 , 7 , 10 , 15 , 20 , 30 } .

Original: The metadata was created by running a grid search on 50 different classiﬁcation data sets, using classiﬁcation accuracy as an objective.
Revised: The metadata was created by running a grid search on 50 different classiﬁcation data sets, using classiﬁcation accuracy as an objective.

Original: Compared to the other two metadata sets, this one is signiﬁcantly smaller.
Revised: Compared to the other two metadata sets, this one is signiﬁcantly smaller.

Original: Its use is mainly motivated by comparing to MetaBO under the same circumstances as used to evaluate MetaBO.
Revised: Its use is mainly motivated by comparing to MetaBO under the same circumstances as used to evaluate MetaBO.

Original: For that reason, we do not use the leave-one-data-set-out cross-validation protocol but instead use the same ﬁxed split created by Volpp et al.
Revised: For that reason, we do not use the leave-one-data-set-out cross-validation protocol but instead use the same ﬁxed split created by Volpp et al.

Original: Statistics about these metadata sets are provided in Table 3.
Revised: Statistics about these metadata sets are provided in Table 3.

Original: D M ETA -L EARNING WITH R EPTILE
Revised: D M ETA -L EARNING WITH R EPTILE

Original: In principle our few-shot surrogate can be combined with any model-agnostic meta-learning approach.
Revised: In principle our few-shot surrogate can be combined with any model-agnostic meta-learning approach.

Original: In this section we compare our proposed meta-learning approach against Reptile (Nichol et al., 2018), a ﬁrst-order approximation of MAML.
Revised: In this section we compare our proposed meta-learning approach against Reptile (Nichol et al., 2018), a ﬁrst-order approximation of MAML.

Original: We use the same hyperparameters and data augmentation as described in Algorithm 1.
Revised: We use the same hyperparameters and data augmentation as described in Algorithm 1.

Original: Reptile introduces a new hyperparameter, an outer learning rate.
Revised: Reptile introduces a new hyperparameter, an outer learning rate.

Original: We set it to 0.1 and decay it linearly to 0.
Revised: We set it to 0.1 and decay it linearly to 0.

Original: As summarized in Table 2, FSBO with the meta-learning approach described in Algorithm 1 is either better or not signiﬁcantly worse than its variation with Reptile.
Revised: As summarized in Table 2, FSBO with the meta-learning approach described in Algorithm 1 is either better or not signiﬁcantly worse than its variation with Reptile.

Original: [Table]
Revised: [Table]