Paper:
Transferring knowledge across tasks to improve data-efficiency is one of the open key challenges in the area of global optimization algorithms. Readily available algorithms are typically designed to be universal optimizers and, thus, often suboptimal for specific tasks. We propose a novel transfer learning method to obtain customized optimizers within the well-established framework of Bayesian optimization, allowing our algorithm to utilize the proven generalization capabilities of Gaussian processes. Using reinforcement learning to meta-train an acquisition function (AF) on a set of related tasks, the proposed method learns to extract implicit structural information and to exploit it for improved data-efficiency. We present experiments on a sim-to-real transfer task as well as on several simulated functions and two hyperparameter search problems. The results show that our algorithm (1) automatically identifies structural properties of objective functions from available source tasks or simulations, (2) performs favourably in settings with both scarse and abundant source data, and (3) falls back to the performance level of general AFs if no particular structure is present.

Review:
Summary: The authors propose a meta-learning based alternative to standard acquisition functions (AFs), whereby a pretrained neural network outputs acquisition values as a function of hand-chosen features. These neural acquisition functions (NAFs) are trained on sets of related tasks using standard RL methods and, subsequently, employed as drop-in replacements for vanilla AFs at test-time. 
Feedback:
Overall, the proposed method makes sense and would benefit from further experimental ablation. Using RL to automatically derive (N)AFs is a nice change of pace from the hand-crafted heuristics that dominate BO. I like the ideas at play here and hope that you will convince me to amend my score.
Results on synthetic functions presented in the body of the paper demonstrate that NAF outperforms, e.g., EI when transferring between homogenous tasks. In contrast, results when transferring between relatively heterogenous functions (Fig. 9) indicate that the aforementioned performance gain reflect NAFs ability to specialize. Two things remain unclear however: 
    a. What types of regularity are NAFs able to exploit?
    b. How quickly do NAFs benefits fall off as tasks become increasingly heterogenous?
Regarding (a), I am not yet convinced that NAFs learn representations that go beyond standard AFs combined with a prior over $x$. To help test this hypothesis, here is a sketch of a simple baseline algorithm:
  1. Fit, e.g., a Gaussian Mixture Model to the top $k=1$ designs $x^{*}_{i}$ on observed tasks $i \in [1, N]$, 
  2. Given a new task $f_{j}$, let log-likelihood $GMM(x)$ act as a 'prior' of sorts on $x$ 
  3. Use cross-validation to tune the scalar parameter $w$ of a new AF defined as the convex combination:
        GMM-UCB(x_{k}) = w * GMM(x_{k}) + (1 - w) * UCB(x_{k})
                                       = w * GMM(x_{k}) + (1 - w) * [\mu_{k} + \sqrt{\beta} * \sigma_{k}].
I suggest using UCB both because NAF could easily learn it from its inputs and because EI values often decay dramatically over the course of BO (I usually set UCB's confidence parameter to a fixed value $\beta = 2$). 
Further simplifying this idea, you could instead use an $\epsilon$-greedy style heuristic that, with probability $\epsilon$, samples without replacement from the set of historical minimizers and otherwise uses a standard AF. These baselines are comparatively straightforward and easily interpreted, so I hope that you will consider adding something along these lines.
Additionally, here are some questions/suggests to help probe (a-b):
  1. Another baseline: EI with multi-task GP? The cubic scaling should be fine for, e.g., 'xxx-20' multi-task variants.
  2. Extend experiments on functions drawn from GP priors (Fig 9):
      i. How does homogeneity (as enforced via the GP hyperprior) impact performance when transferring knowledge?
      ii. Rate of convergence suggests sampled tasks may be too easy; consider using Matern-5/2 and smaller lengthscales [*].
  3. What happens if you expand the task augmentation process to further include, e.g., flips and rotations?
  4. How do 'dimension-agnostic' versions of NAF (where $x$ is excluded from its input) perform on other synthetic tasks?
  5. Visualizing NAF (or the search strategies it produces) would be useful for building intuition.
  6. How were NAF input features chosen? Were alternatives such as also passing the 'best seen' value, considered?
  7. How easy to use are NAFs in comparison to alternative AFs (both in terms of training and test-time maximization)?
  8. Please report regret in log-scale (in appendix); currently, it is hard to tell what is going on in some places. Similarly, the tracked regret level in Figures 3 & 7 changes between tasks without explanation?
In summary, I genuinely want NAF to succeed but am not yet convinced of its performance. If you can provide empirical results to help extinguish my doubts, I will gladly change my assessment.
Nitpicks, Spelling, & Grammar: 
  - Some minor spelling and/or grammatical error, but the paper reads fairly well.
  - On [Chen et al., 2017]:  To the best of my knowledge, these RNN-based methods only require the gradient of the loss function. For example, using GP-based EI as the training signal only requires differentiating through EI + GP rather than through the target function $f$. Similarly, in cases where gradients are not available, the authors elude to use of RL algorithms such as REINFORCE.
[*] For Matern-5/2, just change the prior on your basis functions' weight parameters (https://github.com/metabo-iclr2020/MetaBO/blob/master/metabo/environment/objectives.py#L295) from standard normal to multivariate-t with 5 degrees of freedom.