Paper:
Multi-Task Bayesian Optimization
Bayesian optimization has recently been proposed as a framework for automatically tuning the hyperparameters of machine learning models and has been shown to yield state-of-the-art performance with impressive ease and efficiency. In this paper, we explore whether it is possible to transfer the knowledge gained from previous optimizations to new tasks in order to find optimal hyperparameter settings more efficiently. Our approach is based on extending multi-task Gaussian processes to the framework of Bayesian optimization. We show that this method significantly speeds up the optimization process when compared to the standard single-task approach. We further propose a straightforward extension of our algorithm in order to jointly minimize the average error across multiple tasks and demonstrate how this can be used to greatly speed up k-fold cross-validation. Lastly, we propose an adaptation of a recently developed acquisition function, entropy search, to the cost-sensitive, multi-task setting. We demonstrate the utility of this new acquisition function by leveraging a small dataset to explore hyperparameter settings for a large dataset. Our algorithm dynamically chooses which dataset to query in order to yield the most information per unit cost.

Review:
The paper addresses the problem of automatically 
      selecting hyperparameters for machine learning architectures. The authors 
      consider Bayesian optimization (BO) with Gaussian processes (GP) and adapt 
      this framework to a multi-task setup. This setup allows the sharing of 
      information across multiple optimization problems, and can thus, for 
      instance, exploit knowledge acquired while optimizing the hyperparameters 
      of a learning architecture for one dataset during the optimization of 
      these parameters for a second dataset. They explore three variants of 
      their scheme: (1) transferring knowledge from previously completed 
      optimizations; (2) speeding-up cross-validation by cleverly selecting 
      folds instead of evaluating all parameter settings on all folds; and (3) a 
      cost-sensitive version of hyperparameter optimization that can take 
      advantage of cheaper surrogate tasks (e.g. optimizing the parameters on 
      subsets of the data) for solving a computationally more expensive primary 
      problem by actively selecting the task that is likely to yield the largest 
      improvement per unit cost (e.g. compute time). Optimizing 
      the hyper-parameters of machine learning architectures and algorithms is 
      an important practical problem. The paper falls in line with several 
      recent works that apply sequential model-based optimization (SMBO) 
      techniques, including BO with GPs, to this end (see e.g. refs in the 
      paper). The idea of applying ideas from multi-task learning to BO for 
      hyper-parameter optimization is not completely novel (see the very recent 
      paper by Bardenet et al.; ref [18]), but appears promising and has, to my 
      knowledge, not been explored widely. The paper is generally well 
      written, providing a good explanation of GP BO, and of the proposed 
      extensions. There may be a few places where clarity could be improved (see 
      below). The contribution of the paper is primarily that it brings 
      together, in a sensible manner, various existing tools from the machine 
      learning literature to adapt BO to a multi-task setup (I don't think the 
      "ingredients" themselves are really novel). Furthermore, it proposes -- 
      and empirically explores -- three scenarios for knowledge sharing across 
      tasks in the context of automatic hyper-parameter optimization that are 
      likely to be useful in practice. Specifically, it seems to be 
      building on the work of Snoek et al. (ref [6]), integrating the multi-task 
      kernel of Bonilla et al. (ref [13]), and the recently proposed 
      entropy-search (ES) acquisition function of Hennig & Schuler ([7]). It 
      further finds a new use for the idea of cost-sensitive BO that was 
      previously proposed in [6]. These "architectural" choices seem generally 
      sensible (see below for some additional comments). Of the three 
      application scenarios considered I find especially scenario (3) 
      interesting, and it appears to be a good potential use-case for 
      cost-sensitive BO. I find the experimental evaluation is generally 
      convincing: For each scenario, the authors provide at least one real-world 
      example demonstrating that the respective multi-task approach leads to 
      equally good or better settings of the hyper-parameters in a shorter 
      amount of time than performing BO without taking into account the 
      information from the related task. In some cases additional 
      control experiments could have been performed to provide additional 
      baselines, test the limits of the proposed approach, and to justify and 
      tease apart the effects of the various architectural choices (see below 
      for details). Overall, I think this is a well written paper that 
      is likely to be of practical value for the ML community. 
      Some additional comments: - entropy 
      search criterion (eq. 6): I might well be mistaken but I am a bit confused 
      by the notation, specifically the conditioning of p(f | x, \theta, {x_n, 
      y_n}) on x. If I'm understanding correctly, p(f|...) represents the 
      K-dimensional Gaussian distribution arising from evaluating the posterior 
      over the Gaussian process at points in \tilde{X}. So why do you condition 
      on x, but not on the remaining points of \tilde{X}? Couldn't you just drop 
      x here since you are indexing f in the product term? Also, this 
      seems to be slightly inconsistent with eq. (7) where p(f|x) indicates that 
      the GP is evaluated at a candidate location for an observation (which, if 
      I'm understanding correctly, doesn't necessarily have to coincide with any 
      of the points in \tilde{X})? - I think [7] express your equation 
      (7) in terms of the KL between the P^y_{min} and a uniform distribution. 
      They comment that this is a better choice than the KL you are considering 
      (their page 14). Maybe it'd be worth commenting on this? - 
      two-step heuristic in CV experiments (lines 204ff): Why is this two step 
      approach necessary? Given that the number of folds is probably not huge 
      couldn't you optimize the average objective with respect to an x for each 
      task, and then choose the best x from that set? (Although maybe not much 
      will be gained by this.) - EI vs entropy search (ES; line 240f): 
      You say that EI does not generalize directly to the multi-task setup while 
      ES does. Maybe you could explain why? Couldn't you use the EI on the 
      primary task (per unit cost) as acquisition function? After all, you seem 
      to be using EI on the primary task anyway for the generation of candidate 
      points at which to evaluate the KL-objective (line 261f). - eq. 9: 
      Is there a particular reason why you express (9) in terms of a difference 
      in entropies and not in terms of KL as in (7)? - Experiments in 
      sections 4.1 / 4.2: Am I right to assume that in these experiments you 
      always use the EI criterion as acquisition function? Would you expect ES 
      to perform any differently in these experiments? What about the comparison 
      between STBO and MTBO in section 4.3: how much of the difference is 
      attributable to multi-task vs. single-task and how much to the different 
      acquisition functions? - I assume that for the experiments in 
      4.1/4.2 single-task and multi-task function evaluations take roughly the 
      same amount of time so that there is no need to plot performance as a 
      function of compute time? - Experiments in section 4.1: How do the 
      presented warm-start results compare to very simple baseline strategies 
      for transferring knowledge from previously completed optimizations such as 
      > just using the best-performing hyperparameter setting found in 
      the previous optimization > fixing some the parameters to good 
      values found in the previous optimization and searching only over the 
      remaining ones > searching only over a smaller range for all or 
      some of the parameters based on the results of the previous optimization 
      - Experiments in section 4.1: Is there anything to be gained by 
      considering the results of more than a single completed previous 
      optimization? (E.g. for CNN on SVHN: combining information from CIFAR as 
      well as from the subset of SVHN?) Also, how does the usefulness of 
      information sharing depend on the number of iterations performed in the 
      previous optimization? - Have you considered using alternative 
      multi-task kernels, e.g. one in which task similarity is not modeled 
      "free-form" but depends on certain task features? - Section 4.3: 
      Your experiments seem to be considering a rather simple case of 
      cost-sensitive BO in that the cost exclusively depends on the task s.t. 
      c_t(x) = C_t (or does the cost actually depend on x?), and there are only 
      two tasks. Have you considered other, more challenging scenarios, e.g. the 
      possibility to vary the number of datapoints in the subsets at a finer 
      granularity? - y-axis labels in Fig. 3e,f seem wrong - in 
      eq. (8) shouldn't there be a task label for each past observation e.g. 
      {x_n, y_n, t_n}? - additional reference: Bergstra et al., ICML 
      2013: "Making a Science of Model Search: Hyperparameter OPtimization in 
      Hundreds of Dimensions for Vision Architectures"