  It is commonly believed that Bayesian optimization (BO) algorithms are highly
efficient for optimizing numerically costly functions. However, BO is not often
compared to widely different alternatives, and is mostly tested on narrow sets
of problems (multimodal, low-dimensional functions), which makes it difficult
to assess where (or if) they actually achieve state-of-the-art performance.
Moreover, several aspects in the design of these algorithms vary across
implementations without a clear recommendation emerging from current practices,
and many of these design choices are not substantiated by authoritative test
campaigns. This article reports a large investigation about the effects on the
performance of (Gaussian process based) BO of common and less common design
choices. The experiments are carried out with the established COCO (COmparing
Continuous Optimizers) software. It is found that a small initial budget, a
quadratic trend, high-quality optimization of the acquisition criterion bring
consistent progress. Using the GP mean as an occasional acquisition contributes
to a negligible additional improvement. Warping degrades performance. The
Mat\'ern 5/2 kernel is a good default but it may be surpassed by the
exponential kernel on irregular functions. Overall, the best EGO variants are
competitive or improve over state-of-the-art algorithms in dimensions less or
equal to 5 for multimodal functions. The code developed for this study makes
the new version (v2.1.1) of the R package DiceOptim available on CRAN. The
structure of the experiments by function groups allows to define priorities for
future research on Bayesian optimization.
