Original: No-Regret Approximate Inference via Bayesian Optimisation
Revised: No-Regret Approximate Inference via Bayesian Optimisation

Original: Abstract
Revised: Abstract

Original: We consider Bayesian inference problems where the likelihood function is either expensive to evaluate or only available via noisy estimates.
Revised: We consider Bayesian inference problems where the likelihood function is either expensive to evaluate or only available via noisy estimates.

Original: This setting encompasses application scenarios involving, for example, large datasets or models whose likelihood evaluations require expensive simulations.
Revised: This setting encompasses application scenarios involving, for example, large datasets or models whose likelihood evaluations require expensive simulations.

Original: We formulate this problem within a Bayesian optimisation framework over a space of probability distributions and derive an upper conﬁdence bound (UCB) algorithm to propose non-parametric distribution candidates.
Revised: We formulate this problem within a Bayesian optimisation framework over a space of probability distributions and derive an upper conﬁdence bound (UCB) algorithm to propose non-parametric distribution candidates.

Original: The algorithm is designed to minimise regret, which is deﬁned as the KullbackLeibler divergence with respect to the true posterior in this case.
Revised: The algorithm is designed to minimise regret, which is deﬁned as the KullbackLeibler divergence with respect to the true posterior in this case.

Original: Equipped with a Gaussian process surrogate model, we show that the resulting
Revised: Equipped with a Gaussian process surrogate model, we show that the resulting

Original: UCB algorithm achieves asymptotically no regret.
Revised: UCB algorithm achieves asymptotically no regret.

Original: The method can be easily implemented as a batch
Revised: The method can be easily implemented as a batch

Original: Bayesian optimisation algorithm whose point evaluations are selected via Markov chain Monte Carlo.
Revised: Bayesian optimisation algorithm whose point evaluations are selected via Markov chain Monte Carlo.

Original: Experimental results demonstrate the method’s performance on inference problems.
Revised: Experimental results demonstrate the method’s performance on inference problems.

Original:  INTRODUCTION
Revised:  INTRODUCTION

Original: Bayesian inference problems have been traditionally solved via exact inference methods such as Markov chain Monte
Revised: Bayesian inference problems have been traditionally solved via exact inference methods such as Markov chain Monte

Original: Carlo (MCMC) (Andrieu et al., 2003) when the likelihood function is reasonably cheap to evaluate.
Revised: Carlo (MCMC) (Andrieu et al., 2003) when the likelihood function is reasonably cheap to evaluate.

Original: In these settings, often with the aid of other information, including gradients (Neal, 2011), MCMC methods explore posterior surfaces by performing typically thousands of evaluations until attaining a suitable sample-based posterior approximation.
Revised: In these settings, often with the aid of other information, including gradients (Neal, 2011), MCMC methods explore posterior surfaces by performing typically thousands of evaluations until attaining a suitable sample-based posterior approximation.

Original: As a lightweight alternative variational inference methods (Bishop, 2007) tradeoff approximation accuracy by sampleefﬁciency with methods which can reduce the number of likelihood evaluations by orders of magnitude (Ranganath et al., 2014).
Revised: As a lightweight alternative variational inference methods (Bishop, 2007) tradeoff approximation accuracy by sampleefﬁciency with methods which can reduce the number of likelihood evaluations by orders of magnitude (Ranganath et al., 2014).

Original: However, most state-of-the-art variational inference algorithms rely on gradient information from the probabilistic model within automatic inference frameworks (Salvatier et al., 2016; Bingham et al., 2019).
Revised: However, most state-of-the-art variational inference algorithms rely on gradient information from the probabilistic model within automatic inference frameworks (Salvatier et al., 2016; Bingham et al., 2019).

Original: In some application scenarios in science and engineering, such as geophysical processes (Wellmann et al., 2010), ecological systems (Beaumont, 2010), robotics (Ramos et al., 2019), likelihood evaluations are often expensive or intractable due to the use  complex simulation models of physical phenomena.
Revised: In some application scenarios in science and engineering, such as geophysical processes (Wellmann et al., 2010), ecological systems (Beaumont, 2010) and robotics (Ramos et al., 2019), likelihood evaluations are often expensive or intractable due to the use of complex simulations of physical phenomena.

Revised: In the simulation of high-energy collisions in particle physics, for example, a single event may involve sampling of millions of random variables, making likelihood evaluations completely intractable (Brehmer et al., 2018).

Original: When likelihood functions are not available, one resorts to likelihood-free inference methods, such as approximate
Revised: When likelihood functions are not available, one resorts to likelihood-free inference methods, such as approximate

Original: Bayesian computation (ABC) (Robert, 2016), conditional density estimation (Papamakarios and Murray, 2016), synthetic likelihoods (Ong et al., 2018), etc.
Revised: Bayesian computation (ABC) (Robert, 2016), conditional density estimation (Papamakarios and Murray, 2016), synthetic likelihoods (Ong et al., 2018), etc.

Original: Simulations might still be expensive and limited by resource constraints, imposing challenges to traditional approaches, like ABC.
Revised: Simulations might still be expensive and limited by resource constraints, imposing challenges to traditional approaches, like ABC.

Original: Recent methods address the efﬁciency problem by sequentially learning approximations to the likelihood (Gutmann and
Revised: Recent methods address the efﬁciency problem by sequentially learning approximations to the likelihood (Gutmann and

Original: Corander, 2016; Papamakarios et al., 2019), or the posterior directly (Greenberg et al., 2019), from simulation data.
Revised: Corander, 2016; Papamakarios et al., 2019), or the posterior directly (Greenberg et al., 2019), from simulation data.

Original: In particular, Gutmann and Corander (2016) derive an active learning approach using Bayesian optimisation (BO) (Shahriari et al., 2016) to propose parameters for simulations.
Revised: In particular, Gutmann and Corander (2016) derive an active learning approach using Bayesian optimisation (BO) (Shahriari et al., 2016) to propose simulation parameters.

Original: In this paper, we address the problem of learning posterior approximations via Bayesian optimisation.
Revised: In this paper, we address the problem of learning posterior approximations via Bayesian optimisation.

Original: In particular, we consider settings with black-box likelihoods which might be noisy and expensive to evaluate.
Revised: We consider settings with black-box likelihood functions which might be noisy and expensive to evaluate.

Original: Contrary to the traditional point optimisation for BO, which only seek extreme values, we address the problem of learning the distribution as a whole via non-parametric, sample-based approximations.
Revised: Contrary to the traditional point optimisation for BO, which only seeks extreme values, we address the problem of learning the distribution as a whole via non-parametric, sample-based approximations.

Original: For this purpose, we derive an upper conﬁdence bound (UCB) algorithm to sample approximate posteriors learnt by a Gaussian process (GP) (Rasmussen and Williams, 2006) model on the log-likelihood.
Revised: For this purpose, we derive an upper conﬁdence bound (UCB) algorithm to sample approximate posteriors learnt by a Gaussian process (GP) (Rasmussen and Williams, 2006) model on the log-likelihood.

Original: Compared to previous approaches using BO for approximate inference (Gutmann and Corander, 2016; Acerbi, 2018; Järvenpää et al., 2020), our method proposes a sequence of non-parametric samplebased posterior approximations which allow us to bound theAccepted Kullback-Leibler (KL) divergence between the proposed and the true unknown posterior.
Revised: Compared to previous approaches using GP-based methods for approximate inference (Gutmann and Corander, 2016; Acerbi, 2018; Järvenpää et al., 2020), our method proposes a sequence of non-parametric sample-based posterior approximations which allow us to bound the Kullback-Leibler (KL) divergence between the estimate and the true unknown posterior.

Original: Theoretical results demonstrate that the algorithm is able to converge with sub-linear regret bounds towards the posterior distribution.
Revised: Theoretical results demonstrate that the algorithm is able to converge with sub-linear regret bounds towards the posterior distribution.

Original: We also provide experimental results, assessing the method’s performance in practice on approximate inference problems.
Revised: We also provide experimental results, assessing the method’s performance in practice on approximate inference problems.

Original:  RELATED WORK
Revised:  RELATED WORK

Original: Bayesian optimisation approaches for black-box approximate inference have recently been proposed in the literature.
Revised: Bayesian optimisation approaches for black-box approximate inference have recently been proposed in the literature.

Original: In the context of ABC, Gutmann and Corander (2016) proposed learning a likelihood function from discrepancy evaluations of simulations using a UCB criterion for simulation parameters selection, with other acquisition rules later proposed (Järvenpää et al., 2017).
Revised: In the context of ABC, Gutmann and Corander (2016) proposed learning a likelihood function from discrepancy evaluations of simulations using a UCB criterion for simulation parameters selection, with other acquisition rules later developed (Järvenpää et al., 2017).

Original: Adaptive Bayesian quadrature (ABQ) methods (Kanagawa and Hennig, 2019) are also closely related to BO when considering a search space consisting of probability distributions.
Revised: Adaptive Bayesian quadrature (ABQ) methods (Kanagawa and Hennig, 2019) are also closely related to BO when considering a search space consisting of probability distributions.

Original: Acerbi (2018) presented variational Bayesian Monte
Revised: Acerbi (2018) presented variational Bayesian Monte

Original: Carlo (VBMC), an ABQ framework which sequentially learns variational approximations to a posterior distribution on top of a GP surrogate of the log-joint probability density between parameters and observations.
Revised: Carlo (VBMC), an ABQ framework which sequentially learns variational approximations to a posterior distribution on top of a GP surrogate of the log-joint probability density between parameters and observations.

Original: More recently,
Revised: More recently,

Original: Järvenpää et al.
Revised: Järvenpää et al.

Original: (2020) proposed a GP-based method to sequentially select informative batches of parameters for parallel likelihood evaluations.
Revised: (2020) proposed a GP-based method to sequentially select informative batches of parameters for parallel likelihood evaluations.

Original: Their method selects batch points based on an optimal posterior approximation derived from the current GP log-likelihood estimate.
Revised: Their method selects batch elements based on an optimal posterior approximation derived from the current GP log-likelihood estimate.

Original: Although both ABQ methods do not explicitly involve an optimisation objective, the intermediate posterior approximations at each iteration can be seen as a sequence of candidate solutions optimising an implicit distributional objective.
Revised: Although both ABQ methods do not explicitly involve an optimisation objective, the intermediate posterior approximations at each iteration can be seen as a sequence of candidate solutions optimising an implicit distributional objective.

Original: Non-BO methods based on sequential approximations to a posterior are also plentiful in the likelihood-free inference literature (Papamakarios and Murray, 2016;
Revised: Non-BO methods based on sequential approximations to a posterior are also plentiful in the likelihood-free inference literature (Papamakarios and Murray, 2016;

Original: Papamakarios et al., 2019; Greenberg et al., 2019; Durkan et al., 2020;
Revised: Papamakarios et al., 2019; Greenberg et al., 2019; Durkan et al., 2020;

Original: Cranmer et al., 2020).
Revised: Cranmer et al., 2020).

Original: These methods rely on neural density, or density-ratio, estimators to learn surrogate posterior densities from simulated data.
Revised: These methods rely on neural density, or density-ratio, estimators to learn surrogate posterior densities from simulated data.

Original: Although our framework can also be applied to likelihood-free inference, we do not assume direct access to simulated data, but only to evaluations of a synthetic likelihood or discrepancy function.
Revised: Although our framework can also be applied to likelihood-free inference, we do not assume direct access to simulated data, but only to evaluations of a synthetic likelihood or discrepancy function.

Original: Regarding theoretical results on posterior approximation errors, Kanagawa and Hennig (2019) analyse the convergence rate of adaptive Bayesian quadrature methods, which applies, for example, to Acerbi (2018) and Acerbi (2020).
Revised: Regarding theoretical results on posterior approximation errors, Kanagawa and Hennig (2019) analyse the convergence rate of adaptive Bayesian quadrature methods, which applies, for example, to Acerbi (2018, 2020).

Original: However, in this work, we are concerned with ﬁnite-time upper bounds on the posterior approximation error, deﬁned by the KL divergence.
Revised: However, in this work, we are concerned with ﬁnite-time upper bounds on the posterior approximation error, deﬁned by the KL divergence.

Original: We derive an algorithm which not only comes equipped with such ﬁnite-time guarantees, but which is also shown to achieve asymptotic optimality.
Revised: We derive an algorithm which not only comes equipped with such ﬁnite-time guarantees, but which is also shown to achieve asymptotic optimality.

Original:  PROBLEM FORMULATION
Revised:  PROBLEM FORMULATION

Original: Our goal is to estimate a posterior distribution over parameters θ ∈ Θ ⊂ R d given observed data x ∈ X :
Revised: Our goal is to estimate a posterior distribution over parameters θ ∈ Θ ⊂ R d given observed data x ∈ X :

Original: [Equation]
Revised: [Equation]

Original: [Equation] where we consider a likelihood p ( x | θ ) only available via noisy and/or expensive evaluations, the evidence p ( x ) is assumed intractable, but we have a prior p ( θ ) .
Revised: where we consider a likelihood p ( x | θ ) only available via noisy and/or expensive evaluations, the evidence p ( x ) is assumed intractable, but we have a prior p ( θ ) .

Original: We formulate the posterior estimation in Equation 1 as an optimisation problem over the space P of probability distributions on Θ :
Revised: We formulate the posterior estimation in Equation 1 as an optimisation problem over the space P of probability distributions on Θ :

Original: [Equation]
Revised: [Equation]

Original: where Q ⊂ P is a set of probability distributions over Θ , and D KL ( q || p x ) ≥ 0 denotes the Kullback-Leibler (KL) divergence between q and p x , which is such that D {+KL ( q || p x ) = 
+}
Revised: where Q ⊂ P is a set of probability distributions over Θ , and D KL ( q || p x ) ≥ 0 denotes the Kullback-Leibler (KL) divergence between q and p x , which is such that D {+KL ( q || p x ) = 
+}

Original: [Equation] x .
Revised: ⇐⇒ q = p x .

Original: The problem above can be shown to be equivalent to:
Revised: The problem above can be shown to be equivalent to:

Original: [Equation]
Revised: [Equation]

Original:  where (cid:96) ( θ ) :
Revised: (3) where (cid:96) ( θ ) :

Original: = log p ( x | θ ) denotes the log-likelihood.
Revised: = log p ( x | θ ) denotes the log-likelihood.

Original: The objective function in Equation 3 is known as the evidence lower bound (ELBO), as it lower bounds log p ( x ) .
Revised: The objective function in Equation 3 is known as the evidence lower bound (ELBO), as it lower bounds log p ( x ) .

Original: Finding the global optimum in Equation 3 is usually intractable for an arbitrary log-likelihood function (cid:96) and distributional search space Q .
Revised: Finding the global optimum in Equation 3 is usually intractable for an arbitrary log-likelihood function (cid:96) and distributional search space Q .

Original: In addition, we also consider cases where the estimates of (cid:96) are possibly noisy.
Revised: In addition, we also consider cases where the estimates of (cid:96) are possibly noisy.

Original: Therefore, we make a few regularity assumptions on the log-likelihood function and its estimates.
Revised: Therefore, we make a few regularity assumptions on the log-likelihood function and its estimates.

Original: Regularity assumptions: We assume (cid:96) :
Revised: Regularity assumptions: We assume (cid:96) :

Original: Θ → R lies in a reproducing kernel Hilbert space (RKHS) (Schölkopf and
Revised: Θ → R lies in a reproducing kernel Hilbert space (RKHS) (Schölkopf and

Original: Smola, 2002).
Revised: Smola, 2002).

Original: Given a positive-deﬁnite kernel k : Θ × Θ → R , a RKHS H k is a Hilbert space of functions with inner product (cid:104)· , ·(cid:105) k and norm (cid:107)·(cid:107) k
Revised: Given a positive-deﬁnite kernel k : Θ × Θ → R , a RKHS H k is a Hilbert space of functions with inner product (cid:104)· , ·(cid:105) k and norm (cid:107)·(cid:107) k

Original: = (cid:112) (cid:104)· , ·(cid:105) k , such that f ( θ ) =
Revised: = (cid:112) (cid:104)· , ·(cid:105) k , such that f ( θ ) =

Original: (cid:104) f, k ( · , θ ) (cid:105) k , for any f ∈ H k and any θ ∈ Θ .
Revised: (cid:104) f, k ( · , θ ) (cid:105) k , for any f ∈ H k and any θ ∈ Θ .

Original: We assume k is continuous and bounded with k ( θ , θ ) ≤ 1 , ∀ θ ∈ Θ , and that (cid:107) (cid:96) (cid:107) k ≤ b , where b > 0 is known.
Revised: We assume k is continuous and bounded with k ( θ , θ ) ≤ 1 , ∀ θ ∈ Θ , and that (cid:107) (cid:96) (cid:107) k ≤ b , where b > 0 is known.

Original: Likelihood estimates: We consider a general sequential setting where we may not have direct access to the loglikelihood (cid:96) , but only to noisy estimates ˆ (cid:96) .
Revised: Likelihood estimates: We consider a general sequential setting where we may not have direct access to the loglikelihood (cid:96) , but only to noisy estimates ˆ (cid:96) .

Original: In particular, we assume that the approximation error:
Revised: In particular, we assume that the approximation error:

Original: [Equation]
Revised: [Equation]

Original: is zero-mean σ (cid:15) -sub-Gaussian for a given σ (cid:15) >
Revised: is zero-mean σ (cid:15) -sub-Gaussian for a given σ (cid:15) [Equation]

Original: (Boucheron et al., 2013; Chowdhury and Gopalan, 2017).
Revised: (Boucheron et al., 2013; Chowdhury and Gopalan, 2017).

Original: The sub-Gaussian assumption implies unbiased loglikelihood estimates with an error which is bounded or whose distribution tails decay at least as fast as a Gaussian.
Revised: The sub-Gaussian assumption implies unbiased loglikelihood estimates with an error which is bounded or whose distribution tails decay at least as fast as a Gaussian.

Original: Examples include unbiased synthetic log-likelihood models (Ong et al., 2018) and mini-batching estimates when considering models with bounded log-likelihood functions.
Revised: Examples include unbiased synthetic log-likelihood models (Ong et al., 2018) and mini-batching estimates when considering models with bounded log-likelihood functions.

Original: Regret: Our performance analysis is based on the instant regret of an algorithm choosing a sequence of distributions { q t } t ≥ 1 .
Revised: Regret: Our performance analysis is based on the instant regret of an algorithm choosing a sequence of distributions { q t } t ≥ 1 .

Original: The instant regret at iteration t ≥ 1 is deﬁned as:
Revised: The instant regret at iteration t ≥ 1 is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: If our class of distributions Q is ﬂexible enough, so that p x ∈ Q , then D KL ( q ∗ ||
Revised: If our class of distributions Q is ﬂexible enough, so that p x ∈ Q , then D KL ( q ∗ ||

Original: p x ) = 0 , as q ∗ =
Revised: p x ) = 0 , setting q ∗ =

Original: p x , and:
Revised: p x , and:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: An upper bound on the cumulative regret R T := (cid:80) Tt =1 r t tells us how far the candidates q t picked by the algorithm are to the global optimum q ∗ .
Revised: An upper bound on the cumulative regret R T := (cid:80) Tt =1 r t tells us how far the candidates q t picked by the algorithm are to the global optimum q ∗ .

Original: In particular, the regret for the best candidate up to iteration T is bounded by:
Revised: In particular, the regret for the best candidate up to iteration T is bounded by:

Original: [Equation]
Revised: [Equation]

Original: Therefore, if R T grows sub-linearly with T , the term above vanishes as T → ∞ and the choices of the algorithm get arbitrarily close to the optimal q ∗ in terms of KL divergence with respect to the posterior p x .
Revised: Therefore, if R T grows sub-linearly with T , the term above vanishes as T → ∞ and the choices of the algorithm get arbitrarily close to the optimal q ∗ in terms of KL divergence with respect to the posterior p x .

Original:  BACKGROUND
Revised:  BACKGROUND

Original: This section brieﬂy reviews background on Gaussian processes, Bayesian optimisation via upper conﬁdence bounds, and MCMC algorithms, which we apply for our results.
Revised: This section brieﬂy reviews background on Gaussian processes, Bayesian optimisation via upper conﬁdence bounds, and MCMC algorithms, which we apply for our results.

Original: 4.1 GAUSSIAN PROCESS MODELS
Revised: 4.1 GAUSSIAN PROCESS MODELS

Original: A Gaussian process with mean function m : Θ → R and a positive-deﬁnite covariance function k : Θ × Θ
Revised: A Gaussian process with mean function m : Θ → R and a positive-deﬁnite covariance function k : Θ × Θ

Original: → R represents a prior over functions f : Θ → R so that, for any ﬁnite collection of query points { θ i } Ni =1 ⊂ Θ , the vector f N = [ f ( θ 1 ) , . .
Revised: → R represents a prior over functions f : Θ → R so that, for any ﬁnite collection of query points { θ i } Ni =1 ⊂ Θ , the vector f N = [ f ( θ 1 ) , . .

Original: f ( θ N )]
Revised: f ( θ N )]

Original: T is Gaussian, f N ∼
Revised: T is Gaussian, f N ∼

Original: N ( m N , K N ) , where [ K N ] ij = k
Revised: N ( m N , K N ) , where [ K N ] ij = k

Original: ( θ i , θ j ) .
Revised: ( θ i , θ j ) .

Original: Given a set of observations D N = { ( θ i , z i ) } Ni =1 , where z i = f
Revised: Given a set of observations D N = { ( θ i , z i ) } Ni =1 , where z i = f

Original: ( θ i )
Revised: ( θ i )

Original: + (cid:15) i , and (cid:15)
Revised: + (cid:15) i , and (cid:15)

Original: i ∼ N (0 , η ) , the posterior over f is:
Revised: i ∼ N (0 , η ) , the posterior over f is:

Original: [Equation]
Revised: [Equation]

Original: where:
Revised: where:

Original: [Equation]
Revised: [Equation]

Original: where k N ( θ ∗ ) :
Revised: where k N ( θ ∗ ) :

Original: =
Revised: =

Original: [ k ( θ 1 , θ ∗ ) , . . .
Revised: [ k ( θ 1 , θ ∗ ) , . . .

Original: , k ( θ N , θ )] T .
Revised: , k ( θ N , θ )] T .

Original: To model log-likelihoods, we need to consider that in logscale low-likelihood values tend to −∞ .
Revised: To model log-likelihoods, we need to consider that in logscale low-likelihood values tend to −∞ .

Original: Following previous works (Acerbi, 2018; Järvenpää et al., 2020), we use a negative quadratic mean function and, as covariance function, the squared-exponential k ( θ , θ (cid:48) ) :
Revised: Following previous works (Acerbi, 2018; Järvenpää et al., 2020), we use a negative quadratic mean function and, as covariance function, the squared-exponential kernel:

Original: = σ 2 f exp( − 12 l 2 (cid:107) θ − θ (cid:48) (cid:107) 22 ) , with l, σ f ∈ R , for θ , θ (cid:48)
Revised: [Equation] θ (cid:48) ∈ Θ (12) where l > 0 and σ f > 0 are hyper-parameters (see Rasmussen and Williams, 2006, Ch. 4), and (cid:107)·(cid:107) 2 denotes the

Original: ∈
Revised: Euclidean 2-norm of a vector.

Original: Θ ⊂ R d (Rasmussen and Williams, 2006).
Revised: However, our algorithm and results are not restricted to this particular choice of kernel.

Original: 4.2 THE UPPER CONFIDENCE BOUND ALGORITHM
Revised: 4.2 THE UPPER CONFIDENCE BOUND ALGORITHM

Original: The upper conﬁdence bound (UCB) algorithm has been a popular method in the multi-armed bandits and Bayesian optimisation literature due to its theoretical properties and simplicity (Abbasi-Yadkori et al., 2010; Srinivas et al., 2010;
Revised: The upper conﬁdence bound (UCB) algorithm has been a popular method in the multi-armed bandits and Bayesian optimisation literature due to its theoretical properties and simplicity (Abbasi-Yadkori et al., 2010; Srinivas et al., 2010;

Original: Shahriari et al., 2016; Chowdhury and Gopalan, 2017).
Revised: Shahriari et al., 2016; Chowdhury and Gopalan, 2017).

Original: The algorithm selects queries to solve a sequential optimisation problem by maximising an upper conﬁdence bound over the objective function.
Revised: The algorithm selects queries to solve a sequential optimisation problem by maximising an upper conﬁdence bound over the objective function.

Original: In the case of Gaussian processes, the algorithm selects queries by:
Revised: In the case of Gaussian processes, the algorithm selects queries by:

Original: [Equation]
Revised: [Equation]

Original:  where β t is a parameter which can be adjusted based on the information gain of the GP model, and µ t and σ 2 t deﬁne the
Revised: (13) where β t is a parameter which can be adjusted based on the information gain of the GP model, and µ t and σ 2 t deﬁne the

Original: GP posterior mean and variance, respectively, at time t ≥ 0 .
Revised: GP posterior mean and variance, respectively, at time t ≥ 0 .

Original: UCB algorithms enjoy bounded regret under reasonable assumptions in a point optimisation setting (Abbasi-Yadkori et al., 2010; Durand et al., 2018).
Revised: UCB algorithms enjoy bounded regret under reasonable assumptions in a point optimisation setting (Abbasi-Yadkori et al., 2010; Durand et al., 2018).

Original: In this paper, however, we consider a distribution optimisation setting.
Revised: In this paper, however, we consider a distribution optimisation setting.

Original: 4.3 MARKOV CHAIN MONTE CARLO
Revised: 4.3 MARKOV CHAIN MONTE CARLO

Original: MCMC constitutes a broad class of sampling-based
Revised: MCMC constitutes a broad class of sampling-based

Original: Bayesian inference algorithms, most of which enjoy asymptotic convergence guarantees (Andrieu et al., 2003).
Revised: Bayesian inference algorithms, most of which enjoy asymptotic convergence guarantees (Andrieu et al., 2003).

Original: These methods formulate sampling as a Markov process whose stationary distribution is given by the desired target distribution, i.e. the posterior in Bayesian inference.
Revised: These methods formulate sampling as a Markov process whose stationary distribution is given by the desired target distribution, i.e. the posterior in Bayesian inference.

Original: The main building block of the majority of current MCMC approaches is the
Revised: The main building block of the majority of current MCMC approaches is the

Original: Metropolis-Hastings (MH) algorithm
Revised: Metropolis-Hastings (MH) algorithm

Original: (Hastings, 1970).
Revised: (Hastings, 1970).

Original: In the Bayesian inference formulation, given a proposal distribution π ( θ (cid:48) | θ ) , the MH algorithm accepts or rejects a sample θ (cid:48) given the previous θ with the following acceptance probability:
Revised: In the Bayesian inference formulation, given a proposal distribution π ( θ (cid:48) | θ ) , the MH algorithm accepts or rejects a sample θ (cid:48) given the previous θ with the following acceptance probability:

Original: [Equation]
Revised: [Equation]

Original: Among the most popular choices of MCMC algorithms, we have Hamiltonian Monte Carlo (Neal, 2011) and its related self-adjusting variant, the no-U-turn sampler (NUTS) (Hoffman and Gelman, 2014).
Revised: Among the most popular choices of MCMC algorithms, we have Hamiltonian Monte Carlo (Neal, 2011) and its related self-adjusting variant, the no-U-turn sampler (NUTS) (Hoffman and Gelman, 2014).

Original: In addition, we also have ensemble samplers, which often deal better with multi-modal distributions (Foreman-Mackey et al., 2013).
Revised: In addition, we also have ensemble samplers, which often deal better with multi-modal distributions (Foreman-Mackey et al., 2013).

Original:  DISTRIBUTIONAL BAYESIAN OPTIMISATION
Revised:  DISTRIBUTIONAL BAYESIAN OPTIMISATION

Original: We propose a BO algorithm for approximate Bayesian inference problems (Equation 2) with black-box and/or expensive-to-evaluate likelihood functions.
Revised: We propose a BO algorithm for approximate Bayesian inference problems (Equation 2) with black-box and/or expensive-to-evaluate likelihood functions.

Original: The algorithm iterates between estimating posterior approximations and collecting likelihood evaluations, which are then passed on to update an internal GP model.
Revised: The algorithm iterates between estimating posterior approximations and collecting likelihood evaluations, which are then passed on to update an internal GP model.

Original: In its general form, the algorithm may operate with any class of variational posteriors Q .
Revised: In its general form, the algorithm may operate with any class of variational posteriors Q .

Original: We, however, focus on the class of non-parametric sample-based approximations via MCMC, which offer theoretical and practical beneﬁts, detailed in the next sections.
Revised: We, however, focus on the class of non-parametric sample-based approximations via MCMC, which offer theoretical and practical beneﬁts, detailed in the next sections.

Original: The algorithm is outlined in Algorithm 1.
Revised: The algorithm is outlined in Algorithm 1.

Original: At each iteration, the algorithm selects a variational posterior by maximising a distributional acquisition function consisting of an upper-conﬁdence bound on the ELBO (cf. Equation 3).
Revised: At each iteration, the algorithm selects a variational posterior by maximising a distributional acquisition function consisting of an upper-conﬁdence bound on the ELBO (cf. Equation 3).

Original: From the variational posterior, we sample a batch of S parameters to evaluate the log-likelihood function (cid:96) via its estimator ˆ (cid:96) .
Revised: From the variational posterior, we sample a batch of S parameters to evaluate the log-likelihood function (cid:96) via its estimator ˆ (cid:96) .

Original: The data from the evaluations is fed back to update the algorithm’s internal GP model, and the cycle repeats for a given number of T iterations.
Revised: The data from the evaluations is fed back to update the algorithm’s internal GP model, and the cycle repeats for a given number of T iterations.

Original: In the following, we present the proposed distributional acquisition function and a method to directly sample from its maximiser.
Revised: In the following, we present the proposed distributional acquisition function and a method to directly sample from its maximiser.

Original: 5.1 KULLBACK-LEIBLER UPPER CONFIDENCE BOUND
Revised: 5.1 KULLBACK-LEIBLER UPPER CONFIDENCE BOUND

Original: As the log-likelihood (cid:96) ( θ ) is the only unknown term in
Revised: As the log-likelihood (cid:96) ( θ ) is the only unknown term in

Original: Equation 3, a GP model on (cid:96) :
Revised: Equation 3, a GP model on (cid:96) :

Original: Θ → R allows us to derive an upper conﬁdence bound on the ELBO.
Revised: Θ → R allows us to derive an upper conﬁdence bound on the ELBO.

Original: From this observation, we deﬁne the following acquisition function:
Revised: From this observation, we deﬁne the following acquisition function:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original:  where u t ( θ ) := µ t ( θ )+ β t σ t ( θ ) corresponds to a pointwise
Revised: 15) where u t ( θ ) := µ t ( θ )+ β t σ t ( θ ) corresponds to a pointwise

Original: UCB (Srinivas et al., 2010; Durand et al., 2018), and wederive settings for β t > 0 in Section 6. With this acquisition function, we select candidate distributions as: [Equation] Comparing h in Equation 14 with the main ELBO formulation (Equation 3), we notice that Equation 14 deﬁnes an
Revised: UCB (Srinivas et al., 2010; Durand et al., 2018), and we

Original: ELBO with respect to a surrogate posterior obtained by taking the pointwise UCB u t as a log-likelihood function: [Equation] The equation above deﬁnes a valid probability density function as long as u t is bounded, which is usually the case for practical GP models. We formalise this result in Section 6 (Lemma 2).
Revised: Algorithm

Original: Due to its connection with the KL divergence, we refer to the acquisition function in Equationas Kullback-Leibler upper conﬁdence bound (KL-UCB).

Original: Algorithm 1: KL

Original: UCB 1 D 0 := ∅ 2 for t ∈ { 1 , . . .
Revised: UCB 1 D 0 := ∅ 2 for t ∈ { 1 , . . .

Original: , T } do q t ← argmax q ∈Q h ( q |D t − 1 )
Revised: , T } do q t ← argmax q ∈Q h ( q |D t − 1 )

Original: Sample { θ t,i } Si =1 ∼ q t
Revised: Sample { θ t,i } Si =1 ∼ q t

Original: Evaluate z t,i =
Revised: Evaluate z t,i =

Original: ˆ (cid:96) ( θ t,i ) , i ∈ { 1 , . . .
Revised: ˆ (cid:96) ( θ t,i ) , i ∈ { 1 , . . .

Original: , S }
Revised: , S }

Original: Update GP with D t :
Revised: Update GP with D t :

Original: = D t − 1 ∪ { θ t,i , z t,i } Si = 
Revised: = D t − 1 ∪ { θ t,i , z t,i } Si = derive settings for β t > 0 in Section 6. With this acquisition function, we select candidate distributions as: [Equation] [Equation]

Revised: Comparing h in Equation 15 with the main ELBO formulation (Equation 3), we notice that Equation 15 deﬁnes an

Revised: ELBO with respect to a surrogate posterior obtained by taking the pointwise UCB u t as a log-likelihood function:

Revised: [Equation]

Revised: The equation above deﬁnes a valid probability density function as long as u t is bounded, which is usually the case for practical GP models.

Revised: We formalise this result in Section 6 (Lemma 2).

Revised: Due to its connection with the KL divergence, we refer to the acquisition function in Equationas Kullback-Leibler upper conﬁdence bound (KL-UCB).

Original: 5.2 OPTIMISATION OVER PROBABILITY DISTRIBUTIONS VIA MCMC
Revised: 5.2 OPTIMISATION OVER PROBABILITY DISTRIBUTIONS VIA MCMC

Original: We now address the choice of the class Q of variational posteriors and how to sample from the optimal candidates q t ∈ Q at each BO iteration (cf.
Revised: We now address the choice of the class Q of variational posteriors and how to sample from the optimal candidates q t ∈ Q at each BO iteration (cf.

Original: Equation 15).
Revised: Equation 16).

Original: As discussed in the previous section, KL-UCB as deﬁned in Equationcorresponds to an ELBO with respect to ˆ p t in Equation 16, while also being an upper conﬁdence bound on the main
Revised: As discussed in the previous section, KL-UCB as deﬁned in Equationcorresponds to an ELBO with respect to ˆ p t in Equation 17, while also being an upper conﬁdence bound on the main

Original: ELBO in Equation 3.
Revised: ELBO in Equation 3.

Original: Therefore, for a large enough class of distributions Q , the maximiser q t of Equation 15 is ˆ p t −itself, and we may sample directly from it, bypassing the optimisation step in Equation 15.
Revised: Therefore, for a large enough class of distributions Q , the maximiser q t of Equation 16 is ˆ p t −itself, and we may sample directly from it, bypassing the optimisation step in Equation 16.

Original: To sample from the candidates q t := ˆ p t − 1 at each BO iteration, we need to consider that the denominator in Equation 16 involves a possibly intractable integral.
Revised: To sample from the candidates q t := ˆ p t − 1 at each BO iteration, we need to consider that the denominator in Equation 17 involves a possibly intractable integral.

Original: We address this issue by applying MCMC to sample, which only requires the unnormalised density p ( θ )
Revised: We address this issue by applying MCMC to sample, which only requires the unnormalised density p ( θ )

Original: exp u t − 1 ( θ ) .
Revised: exp u t − 1 ( θ ) .

Original: Due to its asymptotic convergence guarantees (Andrieu et al., 2003), running MCMC with ˆ p t − 1 as our target distribution gives us samples which converge in distribution to ˆ p t − 1 , the optimiser of Equation 15.
Revised: Due to its asymptotic convergence guarantees (Andrieu et al., 2003), running MCMC with ˆ p t − 1 as our target distribution gives us samples which converge in distribution to ˆ p t − 1 , the optimiser of Equation 16.

Original: Therefore, we may see MCMC in this case as a form of optimisation over a space of non-parametric probability distributions.
Revised: We may see MCMC in this case as a form of optimisation over a space of nonparametric probability distributions.

Revised: Further discussions on the topic of MCMC as optimisation are available in the literature of gradient-based methods (Wibisono, 2018).

Original: Also note that we do not need to draw an inﬁnite amount of samples from the MCMC chain.
Revised: Note that we do not need to draw an inﬁnite amount of samples from the MCMC chain.

Original: We only need to run itlong enough so that a batch of S < ∞ samples coming out of the chain approximately follow ˆ p t , i.e. after a  burn-in period or by initialising the chain at a high-probability region, such as a mode of ˆ p t (van Ravenzwaaij et al., 2018).
Revised: We only need to run it so that a batch of S < ∞ samples coming out of the chain approximately follow ˆ p t , i.e. after a long enough burn-in period, or by initialising the chain at a high-probability region, such as a mode of ˆ p t (van Ravenzwaaij et al., 2018).

Original: In our experiments, we use the EMCEE sampler (Foreman-Mackey et al., 2013) which can deal with the multi-modality of the UCB surface at reasonably fast mixing times.
Revised: In our experiments, we use the EMCEE sampler (ForemanMackey et al., 2013) which can deal with the multi-modality of the UCB surface at reasonably fast mixing times.

Original:  THEORETICAL RESULTS
Revised:  THEORETICAL RESULTS

Original: In this section, we present theoretical results on the regret of Algorithm 1 with respect to the objective in Equation 2.
Revised: In this section, we present theoretical results on the regret of Algorithm 1 with respect to the objective in Equation 2.

Original: UCB’s theoretical guarantees are based on uniform concentration bounds.
Revised: UCB’s theoretical guarantees are based on uniform concentration bounds.

Original: In particular, we use the following result presented in Durand et al.
Revised: In particular, we use the following result for point-wise approximations.

Original: Theorem 1 (Durand et al.
Revised: Theorem 1 (Durand et al.

Original: Let k ( θ , θ ) ≤ 1 for all θ ∈ Θ , (cid:107) (cid:96) (cid:107) k ≤ b and (cid:15) t be conditionally σ (cid:15) -sub-Gaussian.
Revised: Let k ( θ , θ ) ≤ 1 for all θ ∈ Θ , (cid:107) (cid:96) (cid:107) k ≤ b and (cid:15) t be conditionally σ (cid:15) -sub-Gaussian.

Original: Then, for any δ ∈ (0 , 1] , with probability at least 1 − δ , uniformly over all t ≥ 0 and θ ∈ Θ , [Equation] where β t ( δ ) =
Revised: Then, for any δ ∈ (0 , 1] , with probability at least 1 − δ , uniformly over all t ≥ 0 and θ ∈ Θ , | (cid:96) ( θ ) − µ t ( θ ) | ≤ β t ( δ ) σ t ( θ ) , where β t ( δ ) =

Original: b + σ (cid:15) (cid:112) 2 η − 1 log( | I + η − 1 K t | 1 / 2 /δ ) .
Revised: b + σ (cid:15) (cid:112) 2 η − 1 log( | I + η − 1 K t | 1 / 2 /δ ) .

Original: Notice that the conﬁdence bound in Theorem 1 is simultaneous over all Θ and is currently the tightest known bound in the UCB literature.
Revised: Notice that the conﬁdence bound in Theorem 1 is simultaneous over all Θ and is currently the tightest known bound in the UCB literature.

Original: In our case, we extend the result above to a concentration bound over the space of probability distributions P with support on Θ .
Revised: In our case, we extend the result above to a concentration bound over the space of probability distributions P with support on Θ .

Original: Lemma 1.
Revised: Lemma 1.

Original: Under our regularity assumptions (cf.
Revised: Under our regularity assumptions (cf.

Original: Section 3), with probability at least 1 − δ , δ ∈ (0 , 1) , we have that the following holds for all t ≥ 0 :
Revised: Section 3), with probability at least 1 − δ , δ ∈ (0 , 1) , we have that the following holds for all t ≥ 0 :

Original: [Equation]
Revised: [Equation]

Original: where β t ( δ ) :
Revised: where β t ( δ ) :

Original: = b + σ (cid:15) (cid:112) 2 η − 1 log( | I + η − 1 K D [Equation] [Equation] [Equation]
Revised: = b + σ (cid:15) (cid:112) 2 η − 1 log( | I + η − 1 K D t | 1 / 2 /δ ) ,

Original: K D :=
Revised: K D :=

Original: [ k ( θ , θ (cid:48) )
Revised: [ k ( θ , θ (cid:48) )

Original: ] θ , θ (cid:48) ∈D represents the GP covariance matrix for a set of observations D , and | A | denotes the determinant of a square matrix A .
Revised: ] θ , θ (cid:48) ∈D represents the GP covariance matrix for a set of observations D , and | A | denotes the determinant of a square matrix A .

Original: Proof sketch.
Revised: Proof sketch.

Original: This result follows by an application of
Revised: This result follows by an application of

Original: Jensen’s inequality to the conﬁdence bound in Durand et al.
Revised: Jensen’s inequality to the conﬁdence bound in Durand et al.

Original: (2018, Theorem 1) for GP-UCB.
Revised: (2018, Theorem 1) for GP-UCB.

Original: The result in Lemma 1 allows us to place a high-probability upper bound on the instant regret of Algorithm 1 when using the acquisition function in Equation 14 and the UCB parameter set to β t := β t ( δ ) .
Revised: The result in Lemma 1 allows us to place a high-probability upper bound on the instant regret of Algorithm 1 when using the acquisition function in Equation 15 and the UCB parameter set to β t := β t ( δ ) .

Original: In addition to the regularity assumptions in Section 3, we need to place conditions on the sampling process { θ i } Si =1 for the bounds to hold.
Revised: In addition to the regularity assumptions in Section 3, we need to place conditions on the sampling process { θ i } Si =1 for the bounds to hold.

Original: We defer full proofs to the supplementary material.
Revised: We defer full proofs to the supplementary material.

Original: The main result depends on the information gain of the GP, which is bounded by:
Revised: The main result depends on the information gain of the GP, which is bounded by:

Original: [Equation]
Revised: [Equation]

Original: where I ( z N , g N | Θ N ) denotes the mutual information between z N = g N + (cid:15) (cid:48) N and g N ∼ N ( 0 , K N ) , with [ K N ] ij =
Revised: where I ( z N , g N | Θ N ) denotes the mutual information between z N = g N + (cid:15) (cid:48) N and g N ∼ N ( 0 , K N ) , with [ K N ] ij =

Original: k ( θ i , θ j ) , θ i , θ j ∈ Θ N and (cid:15) (cid:48)
Revised: k ( θ i , θ j ) , θ i , θ j ∈ Θ N and (cid:15) (cid:48)

Original: N ∼ N ( 0 , η I ) .
Revised: N ∼ N ( 0 , η I ) .

Original: For a compact parameter space Θ , or equivalently when dealing with a compactly supported prior p ( θ ) , the supremum in Equation 18 corresponds to a maximum, which is referred to as the maximum information gain .
Revised: For a compact parameter space Θ , or equivalently when dealing with a compactly supported prior p ( θ ) , the supremum in Equation 19 corresponds to a maximum, which is referred to as the maximum information gain .

Original: For popular choices of kernels, information gain bounds are available in the literature, such as the squared exponential kernel, which is O ((log T ) d +1 )
Revised: For popular choices of kernels, information gain bounds are available in the literature, such as the squared exponential kernel, which is O ((log T ) d +1 )

Original: (Srinivas et al., 2010).
Revised: (Srinivas et al., 2010).

Original: We can now provide the following bound on the cumulative regret of
Revised: We can now provide the following bound on the cumulative regret of

Original: Algorithm 1.
Revised: Algorithm 1.

Original: Theorem 2.
Revised: Theorem 2.

Original: Under the same assumptions in Lemma 1, running Algorithm 1 with h ( q |D t − 1 ) given by Equation 14 and β t := β t ( δ ) , we obtain a bound on the instant regret:
Revised: Under the same assumptions in Lemma 1, running Algorithm 1 with h ( q |D t − 1 ) given by Equation 15 and β t := β t ( δ ) , we obtain a bound on the instant regret:

Original: [Equation]
Revised: [Equation]

Original: and on the cumulative regret at T ≥ 1 :
Revised: and on the cumulative regret at T ≥ 1 :

Original: [Equation]
Revised: [Equation]

Original: which hold with probability at least 1 − δ .
Revised: which hold with probability at least 1 − δ .

Original: Proof sketch.
Revised: Proof sketch.

Original: The ﬁrst result follows by applying the upper conﬁdence bound on (cid:96) to the regret deﬁned in Equationfor an optimal choice according to Equation 15.
Revised: The ﬁrst result follows by applying the upper conﬁdence bound on (cid:96) to the regret deﬁned in Equationfor an optimal choice according to Equation 16.

Original: The second result follows a similar derivation to previous bounds in the literature (cf.
Revised: The second result follows a similar derivation to previous bounds in the literature (cf.

Original: Chowdhury and Gopalan, 2017, Theorem 2).
Revised: Chowdhury and Gopalan, 2017, Theorem 2).

Original: For common choices of kernel functions, such as the squared exponential and the Matérn kernels, the maximum information gain has sub-linear growth (Srinivas et al., 2010).
Revised: For common choices of kernel functions, such as the squared exponential and the Matérn kernels, the maximum information gain has sub-linear growth (Srinivas et al., 2010).

Original: Consequently, Theorem 2 shows that Algorithm 1 achieves vanishing regret under these settings, which allows the algorithm to ﬁnd arbitrarily close approximations to q ∗ within a ﬁnite amount of time.
Revised: Consequently, Theorem 2 shows that Algorithm 1 achieves vanishing regret under these settings, which allows the algorithm to ﬁnd arbitrarily close approximations to q ∗ within a ﬁnite amount of time.

Original: In particular, let us consider the setting where we run exact inference on Equation 15.
Revised: In particular, let us consider the setting where we run exact inference on Equation 16.

Original: Lemma 2.
Revised: Lemma 2.

Original: For any bounded kernel k and bounded GP mean function m , maximising h in Equation 14 at any t ≥is equivalent to:
Revised: For any bounded kernel k and bounded GP mean function m , maximising h in Equation 15 at any t ≥is equivalent to:

Original: [Equation]
Revised: [Equation]

Original: Lemma 2 tells us that the maximiser of h ( q |D t − 1 ) is unique and given by ˆ p t − 1 when ˆ p t − 1 ∈ Q .
Revised: Lemma 2 tells us that the maximiser of h ( q |D t − 1 ) is unique and given by ˆ p t − 1 when ˆ p t − 1 ∈ Q .

Original: In fact, we can sample from ˆ p t − 1 via MCMC, which leads us to the ﬁnal result.
Revised: In fact, we can sample from ˆ p t − 1 via MCMC, which leads us to the ﬁnal result.

Original: Corollary 1.
Revised: Corollary 1.

Original: Under the regularity assumptions in Section 3, with probability greater than 1 − δ , δ ∈ (0 , 1) , we have:
Revised: Under the regularity assumptions in Section 3, with probability greater than 1 − δ , δ ∈ (0 , 1) , we have:

Original: [Equation]
Revised: [Equation]

Original: In particular, for γ T ∈ O ( T α ) with α < 1 / 2 , we have:
Revised: In particular, for γ T ∈ O ( T α ) with α < 1 / 2 , we have:

Original: [Equation]
Revised: [Equation]

Original: This result provides a ﬁnite-sample bound on the KL divergence of posterior approximations (see Appendix C for kernel-speciﬁc bounds) and conﬁrms that the GP-based posterior approximation asymptotically approaches the true posterior p x for the case of popular kernels, such as the squared exponential and the Matérn family, which lead to sub-linear regret bounds (Srinivas et al., 2010).
Revised: This result provides a ﬁnite-sample bound on the KL divergence of posterior approximations (see Appendix C for kernel-speciﬁc bounds) and conﬁrms that the GP-based posterior approximation asymptotically approaches the true posterior p x for the case of popular kernels, such as the squared exponential and the Matérn family, which lead to sub-linear regret bounds (Srinivas et al., 2010).

Original: The result also tells us that a sequence of MCMC samples over a UCB-based likelihood leads us to optimal posterior approximations and a no-regret approximate inference algorithm.
Revised: The result also tells us that a sequence of MCMC samples over a UCB-based likelihood leads us to optimal posterior approximations and a no-regret approximate inference algorithm.

Original: 6.1 CONNECTION WITH OTHER APPROACHES
Revised: 6.1 CONNECTION WITH OTHER APPROACHES

Original: Contrast with point-based approach: In contrast with applying UCB to learn a log-likelihood model via a pointmaximisation problem (Gutmann and Corander, 2016), Theorem 2 establishes a regret bound on distribution choices.
Revised: Contrast with point-based approach: In contrast with applying UCB to learn a log-likelihood model via a pointmaximisation problem (Gutmann and Corander, 2016), Theorem 2 establishes a regret bound on distribution choices.

Original: The distributional regret bound is O ( √ Tγ T + √ γ T γ ST ) , essentially smaller than the regret bound of a GP-UCB algorithm given the same number of likelihood evaluations
Revised: The distributional regret bound is O ( √ Tγ T + √ γ T γ ST ) , essentially smaller than the regret bound of a GP-UCB algorithm given the same number of likelihood evaluations

Original: N = ST , which is O ( √ STγ ST + γ ST ) (Chowdhury and
Revised: N = ST , which is O ( √ STγ ST + γ ST ) (Chowdhury and

Original: Gopalan, 2017) by a factor dependent on the sample batch size S , since γ ST ≥ γ T by monotonicity.
Revised: Gopalan, 2017) by a factor dependent on the sample batch size S , since γ ST ≥ γ T by monotonicity.

Original: In addition, samples from q t consider the likelihood surface as a whole via the expectation within the ELBO, while a pointwise
Revised: In addition, samples from q t consider the likelihood surface as a whole via the expectation within the ELBO, while a pointwise

Original: UCB algorithm would be ultimately concerned with ﬁnding the parameter of maximum likelihood.
Revised: UCB algorithm would be ultimately concerned with ﬁnding the parameter of maximum likelihood.

Original: Connection with VBMC: VBMC (Acerbi, 2018) selects variational posteriors by maximising a lower conﬁdence bound on the ELBO based on GP estimates.
Revised: Connection with VBMC: VBMC (Acerbi, 2018) selects variational posteriors by maximising a lower conﬁdence bound on the ELBO based on GP estimates.

Original: For a GP model on the log-likelihood 2 , VBMC optimises:
Revised: For a GP model on the log-likelihood 2 , VBMC optimises:

Original: [Equation]
Revised: [Equation]

Original: where β ≥ 0 is ﬁxed (e.g. β = 3 ), and σ 2 t − 1 ( q ) :
Revised: where β ≥ 0 is ﬁxed (e.g. β = 3 ), and σ 2 t − 1 ( q ) :

Original: = (cid:82) Θ (cid:82) Θ k t − 1 ( θ , θ (cid:48) )
Revised: = (cid:82) Θ (cid:82) Θ k t − 1 ( θ , θ (cid:48) )

Original: q ( θ ) q ( θ (cid:48) )
Revised: q ( θ ) q ( θ (cid:48) )

Original: d θ d θ (cid:48) corresponds to the variance of the expected log-likelihood.
Revised: d θ d θ (cid:48) corresponds to the variance of the expected log-likelihood.

Original: From the variational posterior q VBMC t that maximises h VBMC ( q |D t − 1 ) , VBMC selects a sequence of points for evaluation: θ VBMC t
Revised: From the variational posterior q VBMC t that maximises h VBMC ( q |D t − 1 ) , VBMC selects a sequence of points for evaluation: θ VBMC t

Original: + i
Revised: + i

Original: [Equation]
Revised: [Equation]

Original: (25) where each point is chosen after updating the GP with the evaluation at the previous point for a total of S VBMC points (set to 5 by default).
Revised: where each point is chosen after updating the GP with the evaluation at the previous point for a total of S VBMC points (set to 5 by default).

Original: VBMC uses a class of non-parametric variational posteriors consisting of mixtures of Gaussians with an adaptible number of components and performs a warm-up run to optimise GP hyper-parameters.
Revised: VBMC uses a class of non-parametric variational posteriors consisting of mixtures of Gaussians with an adaptible number of components and performs a warm-up run to optimise GP hyper-parameters.

Original: Equation 24 corresponds to a lower conﬁdence bound on the ELBO (Acerbi, 2018) in contrast to KL-UCB (cf.
Revised: Equation 25 corresponds to a lower conﬁdence bound on the ELBO (Acerbi, 2018) in contrast to KL-UCB (cf.

Original: Equation 14), which sets an upper conﬁdence bound.
Revised: Equation 15), which sets an upper conﬁdence bound.

Original: Therefore, if the lower conﬁdence bound in VBMC is replaced by KLUCB, the algorithm should enjoy a similar distributional regret  to the one in Theorem 2, since the stochastic process producing the batch does not violate the assumptions in Durand et al.
Revised: Therefore, if the lower conﬁdence bound in VBMC is replaced by KL-UCB, the algorithm should have a similar distributional regret bound to the one in Theorem 2, since the stochastic process producing the batch does not violate the assumptions in Durand et al.

Original: (2018) for Theorem 1 (see also  Chowdhury and Gopalan (2017)).
Revised: (2018) for Theorem 1 (see also the assumptions in Chowdhury and Gopalan (2017)).

Original: 6.2 LIMITATIONS
Revised: 6.2 LIMITATIONS

Original: Our distributional Bayesian optimisation algorithm KLUCB comes with strong theoretical guarantees allowing one not only to learn approximations of a posterior, but also to quantify the error in the approximation.
Revised: Our distributional Bayesian optimisation algorithm KLUCB comes with strong theoretical guarantees allowing one not only to learn approximations of a posterior, but also to quantify the error in the approximation.

Original: However, the algorithm also comes with its own limitations.
Revised: However, the algorithm also comes with its own limitations.

Original: Although not explicit in the regret bounds, the dimensionality of the parameter space Θ affects the bounds via the dependence of the maximum information gain γ T on it.
Revised: Although not explicit in the regret bounds, the dimensionality of the parameter space Θ affects the bounds via the dependence of the maximum information gain γ T on it.

Original: For instance, the squared-exponential kernel k ( θ , θ (cid:48) ) : = σ 2 f exp( − 12 l 2 (cid:107) θ − θ (cid:48) (cid:107) 22 ) , with l, σ f ∈ R + , for θ , θ (cid:48)
Revised: For instance, the squared-exponential kernel (Equation 12) yields γ T ∈ O (log d +1 T )

Original: ∈ Θ ⊂ R d , yields γ T ∈ O (log d +1 T ) (Srinivas et al., 2010), which is exponential in the dimension of the parameter space.
Revised: (Srinivas et al., 2010), which is exponential in the dimension of the parameter space.

Original: In addition, the inversion of the kernel matrix required for GP predictions scales as O ( N 3 ) in time complexity.
Revised: In addition, the inversion of the kernel matrix required for GP predictions scales as O ( N 3 ) in time complexity.

Original: Therefore, our method is more suitable for applications with low-dimensional parameter spaces and where we are limited to a small number of likelihood evaluations.
Revised: Therefore, our method is more suitable for applications with low-dimensional parameter spaces and where we are limited to a small number of likelihood evaluations.

Original: Sparse approximations to GPs (Gijsberts and Metta, 2013; Hensman et al., 2013) and advances in BO methods for high-dimensional search spaces (Wang et al., 2018;
Revised: Sparse approximations to GPs (Gijsberts and Metta, 2013; Hensman et al., 2013) and advances in BO methods for high-dimensional search spaces (Wang et al., 2018;

Original: Mutný and Krause, 2018) may alleviate these issues, but adapting these frameworks to this problem setup is objective of future research.
Revised: Mutný and Krause, 2018) may alleviate these issues, but adapting these frameworks to this problem setup is objective of future research.

Original:  EXPERIMENTS
Revised:  EXPERIMENTS

Original: In this section, we present empirical assessments of the proposed KL-UCB algorithm to complement the theoretical results in the previous section.
Revised: In this section, we present empirical assessments of the proposed KL-UCB algorithm to complement the theoretical results in the previous section.

Original: We start with experiments assessing the theoretical regret bounds in situations where the modelling assumptions hold.
Revised: We start with experiments assessing the theoretical regret bounds in situations where the modelling assumptions hold.

Original: We then follow with experiments comparing KL-UCB against VBMC (Acerbi, 2018) as an adaptive Bayesian quadrature baseline in two test cases where the log-likelihood function is not in the RKHS deﬁned by the GP kernel.
Revised: We then follow with experiments comparing KL-UCB against VBMC (Acerbi, 2018), as an adaptive Bayesian quadrature baseline, in two test cases where the log-likelihood function is not in the RKHS deﬁned by the GP kernel.

Original: Lastly, we present an experiment with a likelihood-free inference problem using a single-point UCB strategy as a baseline (Gutmann and Corander, 2016).
Revised: Lastly, we present an experiment with a likelihood-free inference problem using a single-point UCB strategy as a baseline (Gutmann and Corander, 2016).

Original: For all experiments, KL-UCB was conﬁgured with a standard normal prior and a squared exponential kernel.
Revised: For all experiments, KL-UCB was conﬁgured with a standard normal prior and a squared exponential kernel.

Original: To perform inference over the UCB-based posterior approximation ˆ p t − 1 , we run the ensemble-based MCMC sampler EMCEE (Foreman-Mackey et al., 2013) set with 25 walkers to draw each iteration’s batch of S :
Revised: To perform inference over the UCB-based posterior approximation ˆ p t − 1 , we run the ensemble-based MCMC sampler EMCEE (Foreman-Mackey et al., 2013) set with 25 walkers to draw each iteration’s batch of S :

Original: = 5 after a burn-in period.
Revised: = 5 after a burn-in period.

Original: We set δ := 0 .
Revised: We set δ := 0 .

Original: 2 and adjust the kernel hyper-parameters accordingly.
Revised: 2 and adjust the kernel hyper-parameters accordingly.

Original: Appendix E provides additional experiment details.
Revised: Appendix E provides additional experiment details.

Original: Results are reported in terms of the number of evaluations of the log-likelihood.
Revised: Results are reported in terms of the number of evaluations of the log-likelihood.

Original: As performance metrics, we use both the KL divergence and the Gaussian symmetrised KL divergence (gsKL) used by Acerbi (2018).
Revised: As performance metrics, we use both the KL divergence and the Gaussian symmetrised KL divergence (gsKL) used by Acerbi (2018).

Original: The latter is deﬁned as the symmetric KL divergence between the Gaussian distributions formed by the ﬁrst two moments of the approximate and the true posterior distribution.
Revised: The latter is deﬁned as the symmetric KL divergence between the Gaussian distributions formed by the ﬁrst two moments of the approximate and the true posterior distribution.

Original: In cases where the posterior distribution is not available in closed form, we also run
Revised: In cases where the posterior distribution is not available in closed form, we also run

Original: MCMC and kernel density estimation to recover a reference posterior density.
Revised: MCMC and kernel density estimation to recover a reference posterior density.

Revised: Ablation experiments in the supplementary material analyse the effect of dimensionality in the current framework (see

Revised: Appendix D).

Revised: Code and further ablation experiments are also to be made available online.

Original: 7.1 THEORY ASSESSMENT
Revised: 7.1 THEORY ASSESSMENT

Original: We ﬁrst assess how the theoretical results reﬂect in terms of practical performance in settings where the regularity assumptions are known to hold.
Revised: We ﬁrst assess how the theoretical results reﬂect in terms of practical performance in settings where the regularity assumptions are known to hold.

Original: For this experiment, we[Figure] generate log-likelihood functions in the RKHS H k : 
Revised: For this experiment, we generate log-likelihood functions in the RKHS H k : (cid:96) :

Original: [Equation]
Revised: [Equation]

Original:  where we sample w i ∼ N (0 , 1) , θ i ∼ p ( θ ) , for i ∈ { 1 , . . .
Revised: ) where we sample w i ∼ N (0 , 1) , θ i ∼ p ( θ ) , for i ∈ { 1 , . . .

Original: , M } , with a ﬁxed M := 40 .
Revised: , M } , with a ﬁxed M := 40 .

Original: Figure 1 presents the posterior approximation results for a 1-D RKHS log-likelihood.
Revised: Figure 1 presents the posterior approximation results for a 1-D RKHS log-likelihood.

Original: As seen in Figure 1a, the algorithm is able to recover the modes and generate samples which approximately follow the true posterior. 
Revised: As seen in Figure 1a, the ﬁnal posterior approximation, which uses the GP posterior mean as a log-likelihood, mostly matches the true posterior. The exception is a slight under-estimation of the second, smaller mode.

Original: Although the estimated posterior has narrower modes than the true posterior, this is a well-known effect of using the reverse KL (Wang and Titterington, 2005).
Revised: This, however, is a well-known effect of using the reverse KL, which induces a mode-seeking behaviour (Wang and Titterington, 2005; Bishop, 2007).

Original: In Figure 1b, the regret plot shows that the bound in Theorem 2 is large enough to accommodate for the measured
Revised: In Figure 1b, the regret plot shows that the bound in Theorem 2 is large enough to accommodate for the measured

Original: KL divergence between the proposed distributions and the true posterior.
Revised: KL divergence between the proposed distributions and the true posterior.

Original: As one may observe, the initial regret  is  in general lower. With no data about likelihood evaluations, the initial posterior estimate should reﬂect the prior, a standard normal, which is not far from the generated posteriors.
Revised: The regret bound is calculated based on the formulation of β t in Theorem 1 using the norm of (cid:96) in H k . Details about the KL divergence estimation are available in the supplementary material (see Appendix E). [Figure]

Original: However, regret may temporarily increase as BO samples regions of high model uncertainty.
Revised: [Figure]

Original: 7.2 COMPARISONS
Revised: 7.2 COMPARISONS

Original: In this section, we test KL-UCB on settings where the main assumption (cid:96) ∈
Revised: In this section, we test KL-UCB on settings where the main assumption (cid:96) ∈

Original: H k does not hold.
Revised: H k does not hold.

Original: As a related approximate inference baseline, we use VBMC (Acerbi, 2018).
Revised: As a related approximate inference baseline, we use VBMC (Acerbi, 2018).

Original: Lumpy likelihood: This likelihood function is formulated as a mixture of equally weighted Gaussians: 
Revised: Lumpy likelihood: This likelihood function is formulated as a mixture of equally weighted Gaussians: p (

Original: [Equation]
Revised: [Equation]

Original: ( θ ; θ i , Σ i ) , (27)
Revised: | [Equation] ) =

Original: [Figure]
Revised: [Equation]

Revised: (cid:96)

Original: [Figure]
Revised: [Equation]

Original:  where we sample θ i ∼
Revised: = 1 M M (cid:88) i = N ( θ ; θ i , Σ i ) , (28) where we sample θ i ∼

Original: U [0 , 1] d and Σ i = diag( σ 2 i, 1 , . .
Revised: U [0 , 1] d and Σ i = diag( σ 2 i, 1 , . .

Original: , σ 2 i,d ) with σ i,j ∼
Revised: , σ 2 i,d ) with σ i,j ∼

Original: U [0 . 2 , 0 . 6] , j ∈ { 1 , . . .
Revised: U [0 . 2 , 0 . 6] , j ∈ { 1 , . . .

Original: , d } for a ﬁxed number of components M := 12 , following the setup in Acerbi (2018).
Revised: , d } for a ﬁxed number of components M := 12 , following the setup in Acerbi (2018).

Original: Placing a Gaussian prior p ( θ ) :
Revised: Placing a Gaussian prior p ( θ ) :

Original: = N ( θ ; 0 , σ 2 p I ) with σ p := 0 .
Revised: = N ( θ ; 0 , σ 2 p I ) with σ p := 0 .

Original: 5 on θ , the resulting posterior is available in closed-form as also a mixture of
Revised: 5 on θ , the resulting posterior is available in closed-form as also a mixture of

Original: Gaussians with properly adjusted means and covariances.
Revised: Gaussians with properly adjusted means and covariances.

Original: Figure 2 presents the results for the experiment on the lumpy likelihood function.
Revised: Figure 2 presents the results for the experiment on the lumpy likelihood function.

Original: As we can see, KL-UCB’s performance  is surpassed by that of VBMC.
Revised: As we can see, KL-UCB’s performance [Figure] is surpassed by that of VBMC.

Original: As this problem setting involves a mixture of Gaussians as a likelihood, VBMC has a natural advantage, since this algorithm uses variational posteriors based on mixtures of Gaussians as well.
Revised: As this problem setting involves a mixture of Gaussians as a likelihood, VBMC has a natural advantage, since this algorithm uses variational posteriors based on mixtures of Gaussians as well.

Original: Circular likelihood: We deﬁne a circular 2-dimensional log-likelihood function by:
Revised: Circular likelihood: We deﬁne a circular 2-dimensional log-likelihood function by:

Original: [Equation]
Revised: [Equation]

Original: where θ c := 0 denotes the circle centre, ρ c := 1 . 5 is the circle radius, and s c := 0 .
Revised: where θ c := 0 denotes the circle centre, ρ c := 1 . 5 is the circle radius, and s c := 0 .

Original: 25 is a length-scale.
Revised: 25 is a length-scale.

Original: In Figure 3, we see the performance results for the circular likelihood.
Revised: In Figure 3, we see the performance results for the circular likelihood.

Original: This time KL-UCB noticeably outperforms
Revised: This time KL-UCB noticeably outperforms

Original: VBMC.
Revised: VBMC.

Original: One of the reasons for this outcome is that to be able to cover the narrow circle deﬁned by [Table] [Figure] function, VBMC needs a large number of mixture components (reaching 50 in our tests).
Revised: One of the reasons for this outcome is that to be able to cover the narrow circle deﬁned by the likelihood function, VBMC needs a large number of mixture components (reaching 50 in our tests).

Original: KL-UCB instead samples on top of the non-parametric surface deﬁned by the GP directly.
Revised: KL-UCB instead samples on top of the non-parametric surface deﬁned by the GP directly.

Original: 7.3 LIKELIHOOD-FREE INFERENCE
Revised: 7.3 LIKELIHOOD-FREE INFERENCE

Original: We applied KL-UCB to a likelihood-free inference problem.
Revised: We applied KL-UCB to a likelihood-free inference problem.

Original: In this experiment, we infer the parameters of a classic control environment in OpenAI Gym 3 : the cart-pole.
Revised: In this experiment, we infer the parameters of a classic control environment in OpenAI Gym 4 : the cart-pole.

Original: We ﬁx a given setting for its physics parameters θ real and generate a dataset of 10 trajectories by executing randomly sampled actions.
Revised: We ﬁx a given setting for its physics parameters θ real and generate a dataset of 10 trajectories by executing randomly sampled actions.

Original: Summary statistics γ were the same as Ramos et al.
Revised: Summary statistics ξ were the same as Ramos et al.

Original: (2019), which consist of state-action inner products and the ﬁrst two moments of the trajectory state-differences distribution.
Revised: (2019), which consist of state-action inner products and the ﬁrst two moments of the trajectory state-differences distribution.

Original: The discrepancy was set to ∆ θ := (cid:107) γ θ − γ real (cid:107)  /σ 2 , and we evaluate ˆ (cid:96) ( θ ) := − ∆ θ .
Revised: The discrepancy was set to ∆ θ := (cid:107) ξ θ − ξ real (cid:107) 22 /σ 2 , and we evaluate ˆ (cid:96) ( θ ) := − ∆ θ .

Original: We also placed a Gaussian prior p ( θ ) :
Revised: We also placed a Gaussian prior p ( θ ) :

Original: = N ( θ ; 0 , I ) on the parameters.
Revised: = N ( θ ; 0 , I ) on the parameters.

Original: Our GP model was conﬁgured with the Matérn kernel set with smoothness parameter ν := 32 (Rasmussen and Williams, 2006).
Revised: Our GP model was conﬁgured with the Matérn kernel set with smoothness parameter ν := 3 / 2 (Rasmussen and Williams, 2006).

Original: Other hyper-parameters include the kernel length-scales, signal variance and noise variance, and were  adjusted based on maximum a posteriori estimates from previous runs.
Revised: Other hyper-parameters include the kernel length-scales, signal variance and noise variance, and were [Figure] adjusted based on maximum a posteriori estimates from previous runs.

Original: The GP mean function was set as the log-prior probability density m ( θ ) :
Revised: The GP mean function was set as the log-prior probability density m ( θ ) :

Original: = log p ( θ ), θ ∈ Θ .
Revised: = log p ( θ ) .

Original: We compared KL-UCB against GP-UCB in this experiment.
Revised: We compared KL-UCB against GP-UCB in this experiment.

Original: Both methods were conﬁgured with a ﬁxed β t := 3 , ∀ t ≥ 0 .
Revised: Both methods were conﬁgured with a ﬁxed β t := 3 , ∀ t ≥ 0 .

Original: The main difference between the methods is that GP-UCB sequentially selects single points for evaluations, which maximise the pointwise UCB, while KL-UCB selects an i.i.d.
Revised: The main difference between the methods is that GP-UCB sequentially selects single points for evaluations, which maximise the pointwise UCB, while KL-UCB selects an i.i.d.

Original: batch of points according to the UCB-based posterior approximation.
Revised: batch of points according to the UCB-based posterior approximation.

Original: The GP-UCB approach in this case corresponds to the approach by Gutmann and Corander (2016), providing a baseline as a point-based likelihood-learning method.
Revised: The GP-UCB approach in this case corresponds to the approach by Gutmann and Corander (2016), providing a baseline as a point-based likelihood-learning method.

Original: Figure 4 presents the ﬁnal posterior approximations obtained by KL-UCB and the KL-divergence-based regret for each method.
Revised: Figure 4 presents the ﬁnal posterior approximations obtained by KL-UCB and the KL-divergence-based regret for each method.

Original: As a reference for the "true" posterior, we ran rejection ABC 4 with an adaptable threshold parameter to obtain 1000 samples, and then ﬁt a kernel density estimator to measure the KL divergences.
Revised: As a reference for the "true" posterior, we ran rejection ABC 5 with an adaptable threshold parameter to obtain 1000 samples, and then ﬁt a kernel density estimator to measure the KL divergences.

Original: As the plots show, KL-UCB was able to obtain approximations much closer to the true posterior when compared to GP-UCB.
Revised: As the plots show, KL-UCB was able to obtain approximations much closer to the true posterior when compared to GP-UCB.

Original: It is worth noting that the true posterior’s mass is highly concentrated on a narrow region around the true parameters, making the inference problem relatively hard for a point-by-point method.
Revised: It is worth noting that the true posterior’s mass is highly concentrated on a narrow region around the true parameters, making the inference problem relatively hard for a point-by-point method.

Original: In terms of regret, we measured the KL divergence between the corresponding UCB-based posterior approximation and the true posterior for both methods.
Revised: In terms of regret, we measured the KL divergence between the corresponding UCB-based posterior approximation and the true posterior for both methods.

Original: These results are also in
Revised: These results are also in

Original: Figure 4.
Revised: Figure 4.

Original: Even though GP-UCB and KL-UCB had the same settings for their GP models, the results show that, within the given budget of 200 evaluations of the discrepancy function, GP-UCB was not able to obtain reasonable posterior approximations.
Revised: Even though GP-UCB and KL-UCB had the same settings for their GP models, the results show that, within the given budget of 200 evaluations of the discrepancy function, GP-UCB was not able to obtain reasonable posterior approximations.

Original: Its performance is also more unstable.
Revised: Its performance is also more unstable.

Original:  CONCLUSION
Revised:  CONCLUSION

Original: This paper presented an approach for approximate Bayesian inference via Bayesian optimisation in settings where we need to evaluate black-box likelihood functions.
Revised: This paper presented an approach for approximate Bayesian inference via Bayesian optimisation in settings where we need to evaluate black-box likelihood functions.

Original: The algorithm is composed of an acquisition function formulated as an upper conﬁdence bound over the ELBO of a variational objective.
Revised: The algorithm is composed of an acquisition function formulated as an upper conﬁdence bound over the ELBO of a variational objective.

Original: In contrast to traditional variational inference approaches, the algorithm samples the non-parametric posterior approximation deﬁned by a GP model over the log-likelihood function to propose evaluation points at each iteration.
Revised: In contrast to traditional variational inference approaches, the algorithm samples the non-parametric posterior approximation deﬁned by a GP model over the log-likelihood function to propose evaluation points at each iteration.

Original: The use of a ﬂexible non-parametric class of distributions allows us to propose a sequence of optimal distributional candidates, which theoretical results have shown to lead to a vanishing KL divergence.
Revised: The use of a ﬂexible non-parametric class of distributions allows us to propose a sequence of optimal distributional candidates, which theoretical results have shown to lead to a vanishing KL divergence.

Original: A set of experiments also demonstrated the method in practice.
Revised: A set of experiments also demonstrated the method in practice.

Original: As future work, the method offers a few avenues for improvement, such as possible batch design approaches (Järvenpää et al., 2020) on top of the MCMC samples, scalability to high-dimensional spaces and further theoretical analysis. Author Contributions
Revised: As future work, the method offers a few avenues for improvement, such as possible batch design approaches (Järvenpää et al., 2020) on top of the MCMC samples, scalability to high-dimensional spaces and applications to general optimisation problems.

Original: Rafael Oliveira wrote the paper and developed the idea and results.

Original: Lionel Ott assisted with the initial derivations and further reﬁnements.

Original: Fabio Ramos provided the initial concept of the idea and guided the development of the paper.

Original: APPENDIX
Revised: APPENDIX

Original: This document complements the main paper.
Revised: This appendix complements the main paper with proofs, additional results and experiment details.

Original: In Appendix A, we start with additional theoretical background.
Revised: In Appendix A, we start with additional theoretical background.

Original: Appendix B presents full proofs for the main theoretical results in the paper.
Revised: Appendix B presents full proofs for the main theoretical results in the paper.

Original: Appendix C provides kernel-speciﬁc theoretical upper bounds for the KL divergence of posterior candidates obtained by KL-UCB.
Revised: Appendix C provides kernel-speciﬁc theoretical upper bounds for the KL divergence of posterior candidates obtained by KL-UCB.

Original: In Appendix D, we present a sensitivity analysis on dimensionality effects on KL-UCB.
Revised: In Appendix D, we present a sensitivity analysis on dimensionality effects on KL-UCB.

Original: Finally, in Appendix E, we end with further details on the experiments setup.
Revised: Finally, in Appendix E, we conclude with further details on the experiments setup.

Original: A FURTHER BACKGROUND
Revised: A FURTHER BACKGROUND

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: Let Q and P be two probability measures such that Q is absolutely continuous with respect to P .
Revised: Let Q and P be two probability measures such that Q is absolutely continuous with respect to P .

Original: The Kullback-Leibler (KL) divergence between Q and P is deﬁned as:
Revised: The Kullback-Leibler (KL) divergence between Q and P is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: Case Q and P are both deﬁned on a Euclidean space R d and are absolutely continuous with respect to the Lebesgue measure on this space, the equation above can be rewritten as:
Revised: Case Q and P are both deﬁned on a Euclidean space R d and are absolutely continuous with respect to the Lebesgue measure on this space, the equation above can be rewritten as:

Original: [Equation]
Revised: [Equation]

Original: where q ( θ ) :
Revised: where q ( θ ) :

Original: =
Revised: =

Original: d Q ( θ ) d θ and p ( θ ) :
Revised: d Q ( θ ) d θ and p ( θ ) :

Original: = d P ( θ ) d θ are the probability density functions of Q and P , respectively.
Revised: = d P ( θ ) d θ are the probability density functions of Q and P , respectively.

Original: GP predictive equations: The theoretical results are given in terms of a Gaussian process model GP ( m, k ) for the log-likelihood function (cid:96) :
Revised: GP predictive equations: The theoretical results are given in terms of a Gaussian process model GP ( m, k ) for the log-likelihood function (cid:96) :

Original: Θ → R which is learnt with batches of S i.i.d.
Revised: Θ → R which is learnt with batches of S i.i.d.

Original: samples from a probability distribution q t at each iteration t ∈ { 1 , . .
Revised: samples from a probability distribution q t at each iteration t ∈ { 1 , . .

Original: , T } .
Revised: , T } .

Original: Therefore, at each iteration t ≥ 1 , the GP predictive mean and variance are given by:
Revised: Therefore, at each iteration t ≥ 1 , the GP predictive mean and variance are given by:

Original: [Equation]
Revised: [Equation]

Original:  k t ( θ , θ (cid:48) ) :
Revised: (32) k t ( θ , θ (cid:48) ) :

Original: =
Revised: =

Original: k
Revised: k

Original: [Equation]
Revised: [Equation]

Original: θ
Revised: θ

Original: [Equation]
Revised: [Equation]

Revised: θ

Original: [Equation]
Revised: [Equation]

Original: σ 2 t ( θ ) :
Revised: − k N [Equation] N [Equation]

Original: =
Revised: +

Original: k t ( θ , θ ) ,
Revised: [Equation] N

Original: [Equation]
Revised: [Equation]

Original: 33) where k N t ( θ ) :=
Revised: (33) σ 2 t ( θ ) : =

Revised: k t ( θ , θ ) , (34) where k N t ( θ ) :=

Original: [ k ( θ , θ 1 ) , . . .
Revised: [ k ( θ , θ 1 ) , . . .

Original: , k ( θ , θ N t )] T , [ K N t ] i,j :=
Revised: , k ( θ , θ N t )] T , [ K N t ] i,j :=

Original: k ( θ i , θ j ) , i, j ∈ { 1 , . . .
Revised: k ( θ i , θ j ) , i, j ∈ { 1 , . . .

Original: , N t } , and N t := tS .
Revised: , N t } , and N t := tS .

Original: B PROOFS
Revised: B PROOFS

Original: We here present full proofs for the theoretical results in the paper.
Revised: We here present full proofs for the theoretical results in the paper.

Original: We start with a few auxiliary results and then follow with the proofs for the main results.
Revised: We start with a few auxiliary results and then follow with the proofs for the main results.

Original: B.1 AUXILIARY RESULTS
Revised: B.1 AUXILIARY RESULTS

Original: Lemma 3.
Revised: Lemma 3.

Original: In our settings, the GP posterior variance is always bounded, i.e.:
Revised: In our settings, the GP posterior variance is always bounded, i.e.:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: As the kernel k is bounded, with k ( θ , θ ) ≤ 1 , ∀ θ ∈ Θ , the deﬁnition in Equation 33 leads to:
Revised: As the kernel k is bounded, with k ( θ , θ ) ≤ 1 , ∀ θ ∈ Θ , the deﬁnition in Equation 34 leads to:

Original: [Equation]
Revised: [Equation]

Original: Lemma 4.
Revised: Lemma 4.

Original: Assuming a bounded mean function m : Θ → R , the GP posterior mean is bounded, i.e.:
Revised: Assuming a bounded mean function m : Θ → R , the GP posterior mean is bounded, i.e.:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: As deﬁned in Equation 31, for t ≥ 1 , we have:
Revised: As deﬁned in Equation 32, for t ≥ 1 , we have:

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where we ﬁrst used the fact that ( K N t + η I ) − 1 =
Revised: where we ﬁrst used the fact that ( K N t + η I ) − 1 =

Original: η − 1 ( η − 1 K N t
Revised: η − 1 ( η − 1 K N t

Original: + I ) − 1 (cid:52) η − 1 I due to positive-deﬁniteness, then applied Hölder’s inequality, and ﬁnally the observation that (cid:107) k N t ( θ ) (cid:107) 1 ≤ t , due to the kernel bound.
Revised: + I ) − 1 (cid:52) η − 1 I due to positive-deﬁniteness, then applied Hölder’s inequality, and ﬁnally the observation that (cid:107) k N t ( θ ) (cid:107) 1 ≤ t , due to the kernel bound.

Original: Lemma 5 (Srinivas et al.
Revised: Lemma 5 (Srinivas et al.

Original: (2010, Lemma 5.3)) .
Revised: (2010, Lemma 5.3)) .

Original: The information gain for a sequence of evaluations { θ i , z i } Ni =1 , where z 
Revised: The information gain for a sequence of evaluations { θ i , z i } Ni =1 , where z [Equation] = [Equation]

Revised: i ,

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: N
Revised: N

Original: [Equation]
Revised: [Equation]

Original:  , η ) , can be expressed in terms of the predictive variances.
Revised: 0 , η ) , can be expressed in terms of the predictive variances.

Original: Namely, if f ∼ GP ( m, k ) , then:
Revised: Namely, if f ∼ GP ( m, k ) , then:

Original: [Equation]
Revised: [Equation]

Original: Lemma 6 (Chowdhury and Gopalan (2017, Lemma 4)) .
Revised: Lemma 6 (Chowdhury and Gopalan (2017, Lemma 4)) .

Original: Following the setting of Lemma 5, the sum of predictive standard deviations at a sequence of evaluation points is bounded in terms of the maximum information gain:
Revised: Following the setting of Lemma 5, the sum of predictive standard deviations at a sequence of evaluation points is bounded in terms of the maximum information gain:

Original: [Equation]
Revised: [Equation]

Original: Lemma 7.
Revised: Lemma 7.

Original: Let A ⊂ Θ be a set of points where a function f ∼ GP ( m, k ) was evaluated, so that the GP posterior covariance function and the corresponding variance are given by:
Revised: Let A ⊂ Θ be a set of points where a function f ∼ GP ( m, k ) was evaluated, so that the GP posterior covariance function and the corresponding variance are given by:

Original: [Equation]
Revised: [Equation]

Original: where k ( θ , A ) :=
Revised: where k ( θ , A ) :=

Original: [ k ( θ , a )] a ∈A and K ( A ) :
Revised: [ k ( θ , a )] a ∈A and K ( A ) :

Original: =
Revised: =

Original: [ k ( a , a (cid:48) )] a , a (cid:48) ∈A .
Revised: [ k ( a , a (cid:48) )] a , a (cid:48) ∈A .

Original: Then, for any given set B ⊃ A of evaluations of f , we have:
Revised: Then, for any given set B ⊃ A of evaluations of f , we have:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The result follows by observing that the GP posterior given observations at A is a prior for the GP with the new observations at the complement C := B\A .
Revised: The result follows by observing that the GP posterior given observations at A is a prior for the GP with the new observations at the complement C := B\A .

Original: Then we obtain, for all θ ∈ Θ :
Revised: Then we obtain, for all θ ∈ Θ :

Original: [Equation]
Revised: [Equation]

Original: since k A ( θ , C )( K A ( C ) + η I ) − 1 k A ( C , θ ) is non-negative.
Revised: since k A ( θ , C )( K A ( C ) + η I ) − 1 k A ( C , θ ) is non-negative.

Original: B.2 PROOF OF LEMMA
Revised: B.2 PROOF OF LEMMA

Original: We prove Lemma 1 by applying Jensen’s inequality to Theorem 1.
Revised: We prove Lemma 1 by applying Jensen’s inequality to Theorem 1.

Original: Speciﬁcally, consider that the following holds with probability greater than 1 − δ simultaneously over all Θ :
Revised: Speciﬁcally, consider that the following holds with probability greater than 1 − δ simultaneously over all Θ :

Original: [Equation]
Revised: [Equation]

Original: With the same probability, we then have that:
Revised: With the same probability, we then have that:

Original: [Equation]
Revised: [Equation]

Original: since the absolute value |·| is convex.
Revised: since the absolute value |·| is convex.

Original: Lastly, we note that
Revised: Lastly, we note that

Original: the
Revised: the

Original: GP in Algorithm 1 is taking batches of samples per iteration.
Revised: GP in Algorithm 1 is taking batches of samples per iteration.

Original: Therefore, we have to replace the original K t , which holds t observations, by K D t , which holds all the tS observations collected up to iteration t . {+
+}
Revised: Therefore, we have to replace the original K t , which holds t observations, by K D t , which holds all the tS observations collected up to iteration t . {+
+}

Original: This concludes the proof.
Revised: This concludes the proof.

Original: B.3 PROOF OF THEOREM
Revised: B.3 PROOF OF THEOREM

Original: We prove Theorem 2 in two parts, one for each component of the result.
Revised: We prove Theorem 2 in two parts, one for each component of the result.

Original: In the ﬁrst part we establish a bound on the instant regret.
Revised: In the ﬁrst part we establish a bound on the instant regret.

Original: We then propagate this bound to the cumulative regret.
Revised: We then propagate this bound to the cumulative regret.

Original: For these derivations, we will make use of Lemmaand some of the auxiliary results in Section B.1.
Revised: For these derivations, we will make use of Lemmaand some of the auxiliary results in Section B.1.

Original: Proof of Theorem 2.
Revised: Proof of Theorem 2.

Original: We start by proving that, with h ( q |D t − 1 ) given by KL-UCB and β t :=
Revised: We start by proving that, with h ( q |D t − 1 ) given by KL-UCB and β t :=

Original: β t ( δ ) according to
Revised: β t ( δ ) according to

Original: Theorem 1, we obtain a bound on the instant regret:
Revised: Theorem 1, we obtain a bound on the instant regret:

Original: [Equation]
Revised: [Equation]

Original: By Lemma 1, uniformly over all t ≥ 1 with probability at least 1 − δ , we have that:
Revised: By Lemma 1, uniformly over all t ≥ 1 with probability at least 1 − δ , we have that:

Original: [Equation]
Revised: [Equation]

Original: Applying this bound to the deﬁnition of instant regret yields the ﬁrst part of Theorem 2:
Revised: Applying this bound to the deﬁnition of instant regret yields the ﬁrst part of Theorem 2:

Original: [Equation]
Revised: [Equation]

Original: which holds with probability at least 1 − δ .
Revised: which holds with probability at least 1 − δ .

Original: For the second part
Revised: For the second part

Original: [Equation]
Revised: [Equation]

Original: we apply the bound to the cumulative regret and initally obtain:
Revised: we apply the bound to the cumulative regret and initally obtain:

Original: [Equation]
Revised: [Equation]

Original: since β t ≥ β t − 1 , ∀ t ≥ 1 , and expectations are linear operations.
Revised: since β t ≥ β t − 1 , ∀ t ≥ [Equation] and expectations are linear operations.

Original: Considering the predictive variances above, recall that, at each iteration t ≥
Revised: Considering the predictive variances above, recall that, at each iteration t ≥

Original: [Equation]
Revised: [Equation]

Original: the algorithm selects a batch of i
Revised: the algorithm selects a batch of i

Original: [Equation]
Revised: [Equation]

Original: i.d. points B t := { θ t,i } Si =1 , sampled from q t , where to evaluate the log-likelihood function (cid:96) .
Revised: i.d. points B t := { θ t,i } Si =1 , sampled from q t , where to evaluate the log-likelihood function (cid:96) .

Original: The predictive variance σ 2 t − 1 is conditioned on all previous observations, which are grouped by batches.
Revised: The predictive variance σ 2 t − 1 is conditioned on all previous observations, which are grouped by batches.

Original: We can then decompose, for any t ≥ 1 :
Revised: We can then decompose, for any t ≥ 1 :

Original: [Equation]
Revised: [Equation]

Original: where we use the notation introduced in Lemma
Revised: where we use the notation introduced in Lemma

Original: [Equation]
Revised: [Equation]

Original: and:
Revised: and:

Original: [Equation]
Revised: [Equation]

Original: Therefore, the predictive variance of the batched algorithm is not the same as the predictive variance of a sequential algorithm, and we cannot direcly apply Lemma 6 to bound the last term in Equation 49.
Revised: Therefore, the predictive variance of the batched algorithm is not the same as the predictive variance of a sequential algorithm, and we cannot direcly apply Lemma 6 to bound the last term in Equation 50.

Original: Lemma 7 tells us that the predictive variance given a set of observations is less than the predictive variance given a subset of observations.
Revised: Lemma 7 tells us that the predictive variance given a set of observations is less than the predictive variance given a subset of observations.

Original: Selecting only the ﬁrst point from within each batch and applying Lemma 7, we get, for t ≥ 1 :
Revised: Selecting only the ﬁrst point from within each batch and applying Lemma 7, we get, for t ≥ 1 :

Original: [Equation]
Revised: [Equation]

Original: where Θ t :=
Revised: where Θ t :=

Original: { θ i, 1 } ti =1 , with θ i, 1 ∈ B i , i ∈ { 1 , . .
Revised: { θ i, 1 } ti =1 , with θ i, 1 ∈ B i , i ∈ { 1 , . .

Original: . , t } .
Revised: . , t } .

Original: Note that the right-hand side of the equation above is simply the non-batched GP predictive variance.
Revised: Note that the right-hand side of the equation above is simply the non-batched GP predictive variance.

Original: Furthermore, sample points within a batch are i.i.d., so that θ t, 1 ∼ q t and ˜ θ t ∼ q t are identically distributed.
Revised: Furthermore, sample points within a batch are i.i.d., so that θ t, 1 ∼ q t and ˜ θ t ∼ q t are identically distributed.

Original: We can now apply
Revised: We can now apply

Original: Lemma 6, yielding:
Revised: Lemma 6, yielding:

Original: [Equation]
Revised: [Equation]

Original: Combining this result with Equation 49, we obtain:
Revised: Combining this result with Equation 50, we obtain:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Lastly, from the deﬁnition of β t ( δ ) , we have:
Revised: Lastly, from the deﬁnition of β t ( δ ) , we have:

Original: [Equation]
Revised: [Equation]

Original: where:
Revised: where:

Original: [Equation]
Revised: [Equation]

Original: for g ∼ GP ( m, k ) .
Revised: for g ∼ GP ( m, k ) .

Original: Therefore, the KL-UCB cumulative regret is such that:
Revised: Therefore, the KL-UCB cumulative regret is such that:

Original: [Equation]
Revised: [Equation]

Original: which concludes the proof.
Revised: which concludes the proof.

Original: B.4 PROOF OF LEMMA
Revised: B.4 PROOF OF LEMMA

Original: Lemma 2 states that, for a bounded kernel k , maximising h at any t ≥ 1 is equivalent to:
Revised: Lemma 2 states that, for a bounded kernel k , maximising h at any t ≥ 1 is equivalent to:

Original: [Equation]
Revised: [Equation]

Original: where h is deﬁned as:
Revised: where h is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: with u t ( θ ) :
Revised: with u t ( θ ) :

Original: = µ t ( θ ) + β t σ t ( θ ) , 0 ≤ β t < ∞ , and µ t and σ 2 t deﬁne GP posterior mean and variance at iteration t .
Revised: = µ t ( θ ) + β t σ t ( θ ) , 0 ≤ β t < ∞ , and µ t and σ 2 t deﬁne GP posterior mean and variance at iteration t .

Original: The proof follows by the same argument which turns the general KL divergence minimisation problem into an
Revised: The proof follows by the same argument which turns the general KL divergence minimisation problem into an

Original: ELBO maximisation.
Revised: ELBO maximisation.

Original: The only part to verify is whether ˆ p t − 1 deﬁnes a valid probability density function.
Revised: The only part to verify is whether ˆ p t − 1 deﬁnes a valid probability density function.

Original: Proof of Lemma 2.
Revised: Proof of Lemma 2.

Original: In general, let f : Θ → R be a bounded function on Θ ⊂ R d , i.e. (cid:107) f (cid:107) ∞ < ∞ , and p a probability density function on Θ .
Revised: In general, let f : Θ → R be a bounded function on Θ ⊂ R d , i.e. (cid:107) f (cid:107) ∞ < ∞ , and p a probability density function on Θ .

Original: Then we have that:
Revised: Then we have that:

Original: [Equation]
Revised: [Equation]

Original: Combining Lemma 3 and Lemma 4, we have that u t is always bounded, since:
Revised: Combining Lemma 3 and Lemma 4, we have that u t is always bounded, since:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Then the normalisation constant ζ t for ˆ p t ( θ ) =
Revised: Then the normalisation constant ζ t for ˆ p t ( θ ) =

Original: [Equation]
Revised: [Equation]

Original: p
Revised: p

Original: [Equation]
Revised: [Equation]

Original: θ ) exp u t ( θ ) is bounded, for:
Revised: θ ) exp u t ( θ ) is bounded, for:

Original: [Equation]
Revised: [Equation]

Original: Now
Revised: Now

Original: [Equation]
Revised: [Equation]

Original: for the lemma’s main result
Revised: for the lemma’s main result

Original: [Equation]
Revised: [Equation]

Original: we obtain:
Revised: we obtain:

Original: [Equation]
Revised: [Equation]

Original: which concludes the proof
Revised: which concludes the proof

Original: [Equation]
Revised: [Equation]

Original: B.5 PROOF OF COROLLARY
Revised: B.5 PROOF OF COROLLARY

Original: The corollary is simply a restatement of Theorem 2 in terms of q t = ˆ p t − 1 .
Revised: The corollary is simply a restatement of Theorem 2 in terms of q t = ˆ p t − 1 .

Original: In this case, the KL divergence with respect to the optimal solution is D KL ( q ∗ || p x ) = 0 , since we are considering a class of non-parametric distributions which can recover arbitrary distributions via MCMC.
Revised: In this case, the KL divergence with respect to the optimal solution is D KL ( q ∗ || p x ) = 0 , since we are considering a class of non-parametric distributions which can recover arbitrary distributions via MCMC.

Original: By Theorem
Revised: By Theorem

Original: [Equation]
Revised: [Equation]

Original: we then know that:
Revised: we then know that:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: If γ ST ∈ O ( T α ) , for some α < 1 / 2 , so that α − 1 / 2 < 1 , then:
Revised: If γ ST ∈ O ( T α ) , for some α < 1 / 2 , so that α − 1 / 2 < 1 , then:

Original: [Equation]
Revised: [Equation]

Original: Replacing r t = D KL
Revised: Replacing r t = D KL

Original: (ˆ p t − 1 || p x ) above concludes the proof.
Revised: (ˆ p t − 1 || p x ) above concludes the proof.

Original: Alternative proof.
Revised: Alternative proof.

Original: Another way of proving the same bound on the KL divergence is the following.
Revised: Another way of proving the same bound on the KL divergence is the following.

Original: The KL divergence from ˆ p t to p x is bounded via Lemma 1.
Revised: The KL divergence from ˆ p t to p x is bounded via Lemma 1.

Original: Namely, with probability greater than 1 − δ , we have that:
Revised: Namely, with probability greater than 1 − δ , we have that:

Original: [Equation]
Revised: [Equation]

Original: For the second part of the last term in the right-hand side above
Revised: For the second part of the last term in the right-hand side above

Original: [Equation]
Revised: [Equation]

Original: we also have that:
Revised: we also have that:

Original: [Equation]
Revised: [Equation]

Original: which holds with the same probability as Equation 67.
Revised: which holds with the same probability as Equation 68.

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: we conclude the proof with:
Revised: we conclude the proof with:

Original: [Equation]
Revised: [Equation]

Original: C KL DIVERGENCE BOUNDS FOR KL-UCB WITH SPECIFIC KERNELS
Revised: C KL DIVERGENCE BOUNDS FOR KL-UCB WITH SPECIFIC KERNELS

Original: Corollary 1 connects the regret bound in Theorem 2 with the KL divergence of posterior approximations by KL-UCB when sampling directly from the posterior surrogate induced by UCB.
Revised: Corollary 1 connects the regret bound in Theorem 2 with the KL divergence of posterior approximations by KL-UCB when sampling directly from the posterior surrogate induced by UCB.

Original: In this case, we can bound the KL divergence of the posterior approximations by KL-UCB with respect to the posterior as:
Revised: In this case, we can bound the KL divergence of the posterior approximations by KL-UCB with respect to the posterior as:

Original: [Equation]
Revised: [Equation]

Original: According to Theorem 2, we have R T ∈ O ( γ ST √ T ) .
Revised: According to Theorem 2, we have R T ∈ O ( γ ST √ T ) .

Original: Therefore, to bound the KL divergence, we need kernelspeciﬁc upper bounds for the maximum information gain γ T (Srinivas et al., 2010; Vakili et al., 2021).
Revised: Therefore, to bound the KL divergence, we need kernelspeciﬁc upper bounds for the maximum information gain γ T (Srinivas et al., 2010; Vakili et al., 2021).

Original: In particular, we consider the case of two popular stationary kernel classes.
Revised: In particular, we consider the case of two popular stationary kernel classes.

Original: The ﬁrst one is the squared-exponential kernel: [Equation] Kernel class
Revised: The ﬁrst one is the squared-exponential kernel, used in our

Original: [Table]
Revised: [Table]

Original: where ρ θ ,

Original: [Equation]

Original: =

Original: [Equation] θ − θ [Equation] (cid:107) 2 and l > 0 is a length-scale parameter controlling the smoothness of the functions in the
Revised: experiments.

Original: RKHS. As previously mentioned, this kernel yields a bound γ [Equation] (Srinivas et al., 2010).
Revised: As previously mentioned, this kernel yields a bound γ T ∈ O (cid:16) log d +1 ( T ) (cid:17) (Srinivas et al., 2010).

Original: The second type of kernel is the Matérn class with smoothness parameter ν > 1 / 2 :
Revised: The second type of kernel is the Matérn class with smoothness parameter ν > 1 / 2 :

Original: [Equation]
Revised: [Equation]

Original: (cid:33) ν
Revised: ) where ρ θ , θ (cid:48) :

Original: B ν (cid:32) √ 2 νρ θ , θ (cid:48) l (cid:33) , (72) for θ , θ (cid:48) ∈ Θ , where B ν is the modiﬁed Bessel function of the second kind.
Revised: = (cid:107) θ − θ (cid:48) (cid:107) 2 , l > 0 is a length-scale parameter controlling the smoothness of the functions in the RKHS, Γ is the gamma function, and B ν is the modiﬁed Bessel function of the second kind, for θ , θ (cid:48) ∈ Θ .

Original: This kernel leads to a maximum information gain
Revised: This kernel leads to a maximum information gain

Original: γ T ∈ O (cid:16) T d 2 ν + d log 2 ν 2 ν + d ( T ) (cid:17) according to recent results (Vakili et al., 2021).
Revised: γ T ∈ O (cid:16) T d 2 ν + d log 2 ν 2 ν + d ( T ) (cid:17) according to recent results (Vakili et al., 2021).

Original: Table 1 presents upper bounds for the KL divergence of posterior approximations by KL-UCB.
Revised: Table 1 presents upper bounds for the KL divergence of posterior approximations by KL-UCB.

Original: As the table shows, both kernels lead to an asymptotically vanishing KL divergence in the approximations with respect to the true posterior.
Revised: As the table shows, both kernels lead to an asymptotically vanishing KL divergence in the approximations with respect to the true posterior in general.

Revised: An exception to asymptotic convergence, however, is that convergence does not necessarily hold for d ≥ 2 ν in the case of the Matérn kernel, noticing the exponent in its rate.

Revised: For instance, a Matérn kernel with ν = 3 / 2 would not guarantee convergence in a problem whose parameter space has dimension d > 3 .

Revised: The guarantees for the squaredexponential kernel do not suffer from this drawback, though the rates are possibly worse due to the exponential dependence on d via the logarithmic term.

Revised: In terms of approximation bounds with respect to the number of likelihood evaluations N = ST , we have a KL divergence bound of ˜ O (cid:0) N − 1 / 2 (cid:1) and ˜ O (cid:16) N 1 2 ν + d ( d 2 − ν ) (cid:17) for the squared-exponential and the Matérn kernels, respectively.

Original:  D DIMENSIONALITY EFFECT
Revised: Here the ˜ O -notation suppresses logarithmic factors. Compared to the exponential convergence rates in Kanagawa and Hennig (2019), notice that their results are for a noisefree setting, while we consider settings with (sub-Gaussian) noise. In the noise-free setting, one is usually able to obtain tighter concentration bounds for the GP approximation (see de Freitas et al., 2012). D DIMENSIONALITY EFFECT

Original: In this section, we present a short analysis on the effect of dimensionality on the regret of the KL-UCB algorithm.
Revised: In this section, we present a short analysis on the effect of dimensionality on the regret of the KL-UCB algorithm.

Original: Figure 5 presents the KL-UCB regret for the problem in
Revised: Figure 5 presents the KL-UCB regret for the problem in

Original: Section 7.1 when we increase the dimensionality of the parameter space Θ
Revised: Section 7.1 when we increase the dimensionality of the parameter space Θ

Original: = R d .
Revised: = R d .

Original: As the plot shows, the regret has an exponential dependence on the dimensionality of the parameter space.
Revised: As the plot shows, the regret has an exponential dependence on the dimensionality of the parameter space.

Original: Therefore, the practitioner might need to run the method for longer to obtain reasonable posterior approximations or apply dimensionality reduction methods.
Revised: Therefore, the practitioner might need to run the method for longer to obtain reasonable posterior approximations or apply dimensionality reduction methods.

Original: E DETAILS OF THE EXPERIMENTAL SETUP
Revised: E DETAILS OF THE EXPERIMENTAL SETUP

Original: In this section, we present further details on the experimental setup for the empirical results in the paper.
Revised: In this section, we present further details on the experimental setup for the empirical results in the paper.

Original: In particular, we describe the settings for KL-UCB.
Revised: In particular, we describe the settings for KL-UCB.

Original: For VBMC, we used an implementation provided by its author.
Revised: For VBMC, we used an implementation provided by its author.

Original: KL-UCB setup: For the GP model in KL-UCB, we used
Revised: KL-UCB setup: For the GP model in KL-UCB, we used

Original: GPyTorch (Gardner et al., 2018) with adaptations to perform fast rank-1 Cholesky updates on the GP covariance matrix (see Rasmussen and Williams, 2006, Algorithm 2.1).
Revised: GPyTorch (Gardner et al., 2018) with adaptations to perform fast rank-1 Cholesky updates on the GP covariance matrix (see Rasmussen and Williams, 2006, Algorithm 2.1).

Original: Given the theoretical nature of this work, we did not perform online hyper-parameters learning, contrary to what is usual in other GP-based approximate inference methods (Gutmann and Corander, 2016; Acerbi, 2018).
Revised: Given the theoretical nature of this work, we did not perform online hyper-parameters learning, contrary to what is usual in other GP-based approximate inference methods (Gutmann and Corander, 2016; Acerbi, 2018).

Original: The kernel lengthscale was set as 0.5, a value which provided ﬁtting GP estimates for the generated problem scenarios.
Revised: The kernel lengthscale was set as 0.5, a value which provided ﬁtting GP estimates for the generated problem scenarios.

Original: The GP was conﬁgured as zero mean m := 0 for the RKHS-based problem in Section 7.1, while we used the log-prior probability as the mean function m := log p for the problems in Section 7.2.
Revised: The GP was conﬁgured as zero mean m := 0 for the RKHS-based problem in Section 7.1, while we used the log-prior probability as the mean function m := log p for the problems in Section 7.2.

Original: The latter allows the GP to provide low likelihood estimates for parameters of low prior probability, avoiding excessive exploration of the parameter space.
Revised: The latter allows the GP to provide low likelihood estimates for parameters of low prior probability, avoiding excessive exploration of the parameter space.

Original: In terms of noise settings, observations in the RKHS problem (Section 7.1) were added with Gaussian noise σ (cid:15) := 0 .
Revised: In terms of noise settings, observations in the RKHS problem (Section 7.1) were added with Gaussian noise σ (cid:15) := 0 .

Original: 01 (cid:107) (cid:96) (cid:107) k , while the problems with comparisons against VBMC were conﬁgured with basically no noise ( σ (cid:15) :
Revised: 01 (cid:107) (cid:96) (cid:107) k , while the problems with comparisons against VBMC were conﬁgured with ba- sically no noise ( σ (cid:15) :

Original: = 10 − 6 ), since this algorithm was not originally designed to handle noise (see Acerbi, 2020, for a recent noise-adapted version).
Revised: = 10 − 6 ), since this algorithm was not originally designed to handle noise (see Acerbi, 2020, for a recent noise-adapted version).

Original: The GP noise parameter was correspondingly set as η := 10 − 2 for the RKHS problem, and η := 10 − 4 for the noise-free problems.
Revised: The GP noise parameter was correspondingly set as η := 10 − 2 for the RKHS problem, and η := 10 − 4 for the noise-free problems.

Original: Having a small, non-zero η avoids numerical issues with matrix inversions.
Revised: Having a small, non-zero η avoids numerical issues with matrix inversions.

Original: Lastly, for the setting of β t , the RKHS norm for the loglikelihood function in Section 7.1 is available in closed form, but for the non-RKHS functions we set b := 3 which yields the common 3 standard deviations UCB parameter (also in
Revised: Lastly, for the setting of β t , the RKHS norm for the loglikelihood function in Section 7.1 is available in closed form, but for the non-RKHS functions we set b := 3 which yields the common 3 standard deviations UCB parameter (also in

Original: VBMC (Acerbi, 2018)) in a noiseless setting.
Revised: VBMC (Acerbi, 2018)) in a noiseless setting.

Original: MCMC setup: We conﬁgured the EMCEE sampler (Foreman-Mackey et al., 2013) with 25 walkers and a burnin of 400 samples.
Revised: MCMC setup: We conﬁgured the EMCEE sampler (Foreman-Mackey et al., 2013) with 25 walkers and a burnin of 400 samples.

Original: We selected the S := 5 evaluation points in each BO iteration’s batch out of 500 samples drawn by
Revised: We selected the S := 5 evaluation points in each BO iteration’s batch out of 500 samples drawn by

Original: EMCEE.
Revised: EMCEE.

Original: Sub-sampling from a larger batch of samples reduces the correlation between samples from the chain.
Revised: Sub-sampling from a larger batch of samples reduces the correlation between samples from the chain.

Original: Estimation of KL divergence: To verify theoretical bounds, we measured the KL divergence between KLUCB’s posterior approximations and the true posterior distribution, which is unknown to KL-UCB.
Revised: Estimation of KL divergence: To verify theoretical bounds, we measured the KL divergence between KLUCB’s posterior approximations and the true posterior distribution, which is unknown to KL-UCB.

Original: As the posterior distribution approximations from KL-UCB are sampledbased MCMC estimates, we had to estimate the KL divergence based on samples.
Revised: As the posterior distribution approximations from KL-UCB are sampledbased MCMC estimates, we had to estimate the KL divergence based on samples.

Original: We also took into account that MCMC samples are usually correlated.
Revised: We also took into account that MCMC samples are usually correlated.

Original: To decorrelate the samples, we built a kernel density estimator out of the MCMC samples using a Gaussian kernel with the rule by Scott (1992) for kernel bandwidth selection 7 and then sampled from this continuous density.
Revised: To decorrelate the samples, we built a kernel density estimator out of the MCMC samples using a Gaussian kernel with the rule by Scott (1992) for kernel bandwidth selection 8 and then sampled from this continuous density.

Original: For a large enough number of MCMC samples (2000 for the likelihood-free inference problem, 10000 for the circular likelihood, and 1000 for the other problems), the i.i.d.
Revised: For a large enough number of MCMC samples (2000 for the likelihood-free inference problem, 10000 for the circular likelihood, and 1000 for the other problems), the i.i.d.

Original: samples from the density estimator approximately follow the stationary distribution of the MCMC chain, i.e. its target posterior.
Revised: samples from the density estimator approximately follow the stationary distribution of the MCMC chain, i.e. its target posterior.

Original: We then applied a k-nearest-neighbours method (Szabó, 2014) 8 to obtain the KL divergence estimates.
Revised: We then applied a k-nearest-neighbours method (Szabó, 2014) 9 to obtain the KL divergence estimates.