Original: Tree ensemble kernels for Bayesian optimization with known constraints over mixed-feature spaces
Revised: Tree ensemble kernels for Bayesian optimization with known constraints over mixed-feature spaces

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Tree ensembles can be well-suited for black-box optimization tasks such as algorithm tuning and neural architecture search, as they achieve good predictive performance with little or no manual tuning, naturally handle discrete feature spaces, andare relatively insensitive to outliers in the training data.
Revised: Tree ensembles can be well-suited for black-box optimization tasks such as algorithm tuning and neural architecture search, as they achieve good predictive performance with little or no manual tuning, naturally handle discrete feature spaces, and are relatively insensitive to outliers in the training data.

Original: Two well-known challengesin using tree ensembles for black-box optimization are (i) effectively quantifyingmodel uncertainty for exploration and (ii) optimizing over the piece-wise constantacquisition function.
Revised: Two well-known challenges in using tree ensembles for black-box optimization are (i) effectively quantifying model uncertainty for exploration and (ii) optimizing over the piece-wise constant acquisition function.

Original: To address both points simultaneously, we propose using thekernel interpretation of tree ensembles as a Gaussian Process prior to obtain modelvariance estimates, and we develop a compatible optimization formulation for theacquisition function.
Revised: To address both points simultaneously, we propose using the kernel interpretation of tree ensembles as a Gaussian Process prior to obtain model variance estimates, and we develop a compatible optimization formulation for the acquisition function.

Original: The latter further allows us to seamlessly integrate knownconstraints to improve sampling efficiency by considering domain-knowledgein engineering settings and modeling search space symmetries, e.g. hierarchicalrelationships in neural architecture search.
Revised: The latter further allows us to seamlessly integrate known constraints to improve sampling efficiency by considering domain-knowledge in engineering settings and modeling search space symmetries, e.g., hierarchical relationships in neural architecture search.

Original: Our framework performs as well asstate-of-the-art methods for unconstrained black-box optimization over continuous/discrete features and outperforms competing methods for problems combiningmixed-variable feature spaces and known input constraints.
Revised: Our framework performs as well as state-of-the-art methods for unconstrained black-box optimization over continuous/discrete features and outperforms competing methods for problems combining mixed-variable feature spaces and known input constraints.

Original:  Introduction
Revised:  Introduction

Original: Many black-box optimization problems contain feature relationships known a priori based on domainknowledge, such as hierarchies or constraints [40].
Revised: Many black-box optimization problems contain feature relationships known a priori based on domain knowledge, such as hierarchies or constraints [51, 68].

Original: For example, hierarchical structures arise inneural architecture search [15, 58], where hyperparameters such as kernel size are only relevant if aconvolutional layer is selected.
Revised: For example, hierarchical structures arise in neural architecture search [21, 72], where hyperparameters such as kernel size are only relevant if a convolutional layer is selected.

Original: Explicit constraints can also arise, e g. matching kernel size and strideto input channel size and padding.
Revised: Explicit constraints can also arise, e g., matching kernel size and stride to input channel size and padding.

Original: In many cases, Bayesian optimization can incorporate knownhierarchies and/or constraints, given that a suitable surrogate model is selected.
Revised: In many cases, Bayesian optimization can incorporate known hierarchies and/or constraints, given that a suitable surrogate model is selected.

Original: To this end, Fromontet al.
Revised: To this end, Fromont et al.

Original: [17] and Nijssen and Fromont [41] impose a variable hierarchy by constraining the splittingorder of decision trees, i.e. certain attributes must be selected before others.
Revised: [23] and Nijssen and Fromont [53] impose a variable hierarchy by constraining the splitting order of decision trees, i.e., certain attributes must be selected before others.

Original: Tree-based models, such as random forests or gradient-boosted trees, remain popular in many applications, as they inherit the innate ability of simple decision trees to seamlessly handle categorical anddiscrete input spaces.
Revised: Tree-based models, such as random forests or gradient-boosted trees, remain popular in many applications, as they inherit the innate ability of simple decision trees to seamlessly handle categorical and discrete input spaces.

Original: Moreover, they are highly parallelizable and scalable to high-dimensional data.
Revised: Moreover, they are highly parallelizable and scalable to high-dimensional data.

Original: Despite these modelling advantages, the deployment of tree-based models in Bayesian optimization has been limited by challenges pertaining to (i) quantifying prediction uncertainty and (ii) optimizingacquisition functions defined by their discontinuous response surfaces [49].
Revised: Despite these modelling advantages, the deployment of tree-based models in Bayesian optimization has been limited by challenges pertaining to (i) quantifying prediction uncertainty and (ii) optimizing acquisition functions defined by their discontinuous response surfaces [61].

Original: Early works, e.g. thepopular SMAC algorithm [27], addressed (i) using empirical variance within a tree ensemble and (ii)via local and/or random search methods.
Revised: Early works, e.g., the popular SMAC algorithm [34], addressed (i) using empirical variance within a tree ensemble and (ii) via local and/or random search methods.

Original: Moreover, recent works [38, 39, 53] propose mixed-integerformulations for tree ensembles, enabling optimization over their mean functions.
Revised: Moreover, recent works [49, 50, 66] propose mixed-integer formulations for tree ensembles, enabling optimization over their mean functions.

Original: Contributions.
Revised: Contributions.

Revised: Sections 2 and 3 present related work and methods used to derive the approach proposed in this paper.

Original: We present a mixed-integer second-order cone optimization formulation for tree kernel Gaussian processes in Section 4 and show that that the tree agreement ratio , i.e. the hyper- parameter introduced by the tree ensemble kernel, sufficiently represents the model uncertaintyin Section 5.1.
Revised: We present a mixed-integer second-order cone optimization formulation for tree kernel Gaussian processes in Section 4 and show that that the tree agreement ratio , i.e., the hyperparameter introduced by the tree ensemble kernel, sufficiently represents the model uncertainty in Section 5.1.

Original: Section 5.2 shows that solving the mixed-integer second-order cone optimizationproblem considerably outperforms sampling-based strategies in Bayesian optimization.
Revised: Section 5.2 shows that solving the mixed-integer second-order cone optimization problem considerably outperforms sampling-based strategies in Bayesian optimization.

Original: Our approachof using tree ensemble kernels as a Gaussian process prior is particularly useful for applicationscombining mixed-variable spaces and known input constraints.
Revised: Our approach of using tree ensemble kernels as a Gaussian process prior is particularly useful for applications combining mixed-variable spaces and known input constraints.

Revised: A Python implementation of the proposed algorithm is available at: www.github.com/cog-imperial/tree_kernel_gp

Original:  Related work
Revised:  Related work

Original: Bayesian optimization (BO) solves [16, 20, 49]: x ∗ f ∈ arg min x f ( x ) , where f is an expensive-toevaluate black-box function that can be queried at inputs x ∈ X to derive the optimal solution x ∗ f .
Revised: Bayesian optimization (BO) solves [22, 27, 61]: x ∗ f ∈ arg max x f ( x ) , where f is an expensive-toevaluate black-box function that can be queried at inputs x ∈ X to derive the optimal solution x ∗ f .

Original: BO iteratively updates a surrogate model of f and optimizes a corresponding acquisition function thatbalances exploitation and exploration.
Revised: BO iteratively updates a surrogate model of f and optimizes a corresponding acquisition function that balances exploitation and exploration.

Original: Minimizing the acquisition function produces a new query x ∗which is evaluated and added to the set of observations.
Revised: Maximizing the acquisition function produces a new query x ∗ which is evaluated and added to the set of observations.

Original: Gaussian processes (GPs) [45] are a commonchoice of BO surrogate due to their flexibility, e.g. domain-specific knowledge can be built into the
Revised: Gaussian processes (GPs) [57] are a common choice of BO surrogate due to their flexibility, e.g., domain-specific knowledge can be built into the

Original: GP prior via mean and kernel functions, and reliable uncertainty quantification to identify unexploredsearch areas.
Revised: GP prior via mean and kernel functions, and reliable uncertainty quantification to identify unexplored search areas.

Original: Open-source tools like BoTorch [1] implement BO with GP surrogates and offer a wideselection of kernels mainly suited for continuous search spaces.
Revised: Open-source tools such as BoTorch [2] implement BO with GP surrogates and offer a wide selection of kernels mainly suited for continuous search spaces.

Original: Prior works developed GPs withmodified kernels to integrate discrete features [6, 14, 19, 23, 47] or considered conditional featurespaces [22, 28, 35, 37].
Revised: Prior works developed GPs with modified kernels to integrate discrete features [10, 19, 25, 30, 59] or considered conditional feature spaces [29, 35, 43, 48]. Nguyen et al.

Revised: [52] integrate catgorical and category-specific continuous inputs by formulating the black-box optimization problem as a multi-arm bandit problem for which each category corresponds to an arm.

Revised: Similarly, Gopakumar et al.

Revised: [26] handle mixed-type inputs using multi-armed bandits.

Original: Besides GPs, tree ensemble-based surrogates show excellent performance forblack-box optimization with mixed-variable settings, i.e. with continuous, integer and categoricalvariables, and for structured search spaces, e.g. hierarchical and conditional feature spaces [27].
Revised: Besides GPs, tree ensemble-based surrogates show excellent performance for black-box optimization with mixed-variable settings, i.e., with continuous, integer and categorical variables, and for structured search spaces, e.g., hierarchical and conditional feature spaces [34].

Original: Black-box optimization tools using tree ensembles, e.g. SMAC [27] and Scikit-Optimize ( SKOPT ) [24],are useful for applications such as neural architecture search (NAS) and algorithm tuning.
Revised: Black-box optimization tools using tree ensembles, e.g., SMAC [34] and Scikit-Optimize ( SKOPT ) [31], are useful for applications such as neural architecture search (NAS) and algorithm tuning.

Original: Shahriariet al.
Revised: Shahriari et al.

Original: [49] mention challenges in deploying tree ensembles for BO: (i) quantifying uncertainty forexploration purposes, and (ii) optimizing over the non-differentiable discrete acquisition functionto determine the next query point.
Revised: [61] mention challenges in deploying tree ensembles for BO: (i) quantifying uncertainty for exploration purposes, and (ii) optimizing over the non-differentiable discrete acquisition function to determine the next query point.

Original: SMAC identifies uncertain search space regions using empiricalvariables across tree predictions of the random forest and optimizes the acquisition function combininglocal and random search.
Revised: SMAC identifies uncertain search space regions using empirical variables across tree predictions of the random forest and optimizes the acquisition function combining local and random search.

Original: Bergstra et al.
Revised: Bergstra et al.

Original: [3] proposes the Tree Parzen Estimator (TPE) to handlecategorical variables and conditional structures by modeling individual input dimensions by a kerneldensity estimator.
Revised: [5] proposes the Tree Parzen Estimator (TPE) to handle categorical variables and conditional structures by modeling individual input dimensions by a kernel density estimator.

Original: However, the TPE approach ignores dependencies between dimensions.
Revised: However, the TPE approach ignores dependencies between dimensions.

Original: Forgradient-boosted tree ensembles , SKOPT derives uncertainty with quantile regression to fit two modelsfor the 16th and 84th percentile and averages the predictions to estimate the standard deviation.
Revised: For gradient-boosted tree ensembles, SKOPT derives uncertainty with quantile regression to fit two models for the 16th and 84th percentile and averages the predictions to estimate the standard deviation.

Original: Forrandom forests, SKOPT uses an uncertainty strategy similar to SMAC .
Revised: For random forests, SKOPT uses an uncertainty strategy similar to SMAC .

Original: In general, SKOPT relies onrandom sampling to optimize the acquisition function.
Revised: In general, SKOPT relies on random sampling to optimize the acquisition function.

Original: Miši´c [38] proposed a mixed-integer optimization formulation for tree ensemble mean functionsthat has been used in several applications [7, 39, 53, 54].
Revised: Miši´c [49] proposed a mixed-integer optimization formulation for tree ensemble mean functions that has been used in several applications [11, 50, 65–67].

Original: Besides improving the solution to anacquisition function, mixed-integer formulations also allow explicit consideration of input constraintsto incorporate domain-knowledge.
Revised: Besides improving the solution to an acquisition function, mixed-integer formulations also allow explicit consideration of input constraints to incorporate domain knowledge.

Original: Some software tools, e.g. BoTorch, support linear equality andinequality constraints of continuous variables at the acquisition function optimization step while treeensemble-based algorithms do not support input constraints.
Revised: Some software tools, e.g., BoTorch, support linear equality and inequality constraints of continuous variables at the acquisition function optimization step, while tree ensemble-based algorithms do not support input constraints.

Original: Papalexopoulos et al.
Revised: Papalexopoulos et al.

Original: [42] use ReLUneural networks as surrogate models and deploy a mixed-integer linear formulation to optimize theacquisition function.
Revised: [54] use ReLU neural networks as surrogate models and deploy a mixed-integer linear formulation to optimize the acquisition function.

Original: The approach relies on random initialization and stochasticity in the modeltraining to allow for exploration.
Revised: The approach relies on random initialization and stochasticity in the model training to allow for exploration.

Original: Daxberger et al.
Revised: Daxberger et al.

Original: [11] handles mixed-variable search spaces by usinga Bayesian linear regressor that uses an integer solver to search the discrete subspace.
Revised: [16] handle mixed-variable search spaces by using a Bayesian linear regressor that uses an integer solver to search the discrete subspace.

Original: The authorsintroduce features capturing the discrete parts of the search space by using a BOCS model [2, 13]while continuous parts are handled with random Fourier features [44].
Revised: The authors introduce features capturing the discrete parts of the search space by using a BOCS model [3, 18], while continuous parts are handled with random Fourier features [56].

Original: Genetic Algorithms ( GA ) areanother class of algorithms, which deploy evolution-based selection heuristics to minimize black-boxfunctions [29].
Revised: Genetic Algorithms ( GA ) are another class of algorithms, which deploy evolution-based selection heuristics to maximize black-box functions [36].

Original: While there is no feasibility guarantee for input constraints, GA implementations likepymoo [4] support constraint optimization by minimizing constraint violation.
Revised: While there is no feasibility guarantee for input constraints, GA implementations like pymoo [7] support constraint optimization by minimizing constraint violation.

Original: We compare our BO approach, which uses the kernel interpretation of tree ensembles as a Gaussianprocess prior, to: SMAC , the random forest and gradient-boosted tree versions of SKOPT , i.e. SKOPT-RF ,and SKOPT-GBRT ,  in the Section 5 numerical studies as a baseline for other tree ensemble-basedalgorithms.
Revised: We compare our BO approach, which uses the kernel interpretation of tree ensembles as a Gaussian process prior, to: SMAC , the random forest and gradient-boosted tree versions of SKOPT , ( SKOPT-RF , and SKOPT-GBRT , respectively) in the Section 5 numerical studies as a baseline for other tree ensemble-based algorithms.

Original: We also compare against the default upper-confidence bound and expected improvement BO implementations of BoTorch, i.e. UCB-MATERN and EI-MATERN ,  and the default GA algorithm ofpymoo to include black-box algorithms that partially support constraint optimization.[Figure]
Revised: We also compare against the default upper-confidence bound and expected improvement BO implementations of BoTorch ( UCB-MATERN and EI-MATERN , respectively) and the default GA algorithm of pymoo to include black-box algorithms that partially support constrained optimization.

Original:  Technical background on prior work3.1 Tree ensemble kernel as a Gaussian process prior
Revised:  Technical background on prior work 3.1 Tree ensemble kernel as a Gaussian process prior

Original: Exploring the search space requires quantifying the uncertainty of the underlying surrogate model.
Revised: Exploring the search space requires quantifying the uncertainty of the underlying surrogate model.

Original: We use the kernel interpretation of tree ensembles based on random partitions [10, 59].
Revised: We use the kernel interpretation of tree ensembles based on random partitions [15, 73].

Original: The treekernel captures correlation between two input data points ( x , x ′ ) ∈ R n :
Revised: The tree kernel captures correlation between two input data points ( x , x ′ ) ∈ R n :

Original: [Equation]
Revised: [Equation]

Original: To derive the tree kernel, we first train a gradient boosted tree ensemble T on data set X ∈ R m × nwith n denoting the dimensionality of the search space and m the size of the data set.
Revised: To derive the tree kernel, we first train a gradient boosted tree ensemble T on data set X ∈ R m × n with n denoting the dimensionality of the search space and m the size of the data set.

Original: Every tree t inthe tree ensemble T maps inputs x ∈ R n onto a leaf l by sequentially evaluating splitting conditions.
Revised: Every tree t in the tree ensemble T maps inputs x ∈ R n onto a leaf l by sequentially evaluating splitting conditions.

Original: Each leaf l defines a subspace x l ⊂ R n restricted by active splits s ∈ splits ( t ) .
Revised: Each leaf l defines a subspace x l ⊂ R n restricted by active splits s ∈ splits ( t ) .

Original: Two inputs ( x , x ′ )are fully correlated in tree t if both end up in the leaf subspace x t,l and uncorrelated otherwise.
Revised: Two inputs ( x , x ′ ) are fully correlated in tree t if both end up in the leaf subspace x t,l and uncorrelated otherwise.

Original: The
Revised: The

Original: Eq.
Revised: Eq.

Original: (1) vector z ( x ) consists of binary elements z t,l indicating if leaf l ∈ L t is active for input x , with
Revised: (1) vector z ( x ) consists of binary elements z t,l indicating if leaf l ∈ L t is active for input x , with

Original: L t denoting the set of all leaves in tree t .
Revised: L t denoting the set of all leaves in tree t .

Original: The inner product z ( x ) ⊺ z ( x ′ ) , normalized by the totalnumber of trees |T | , gives the ratio of trees in the ensemble for which x and x ′ fall into the same leaf.
Revised: The inner product z ( x ) ⊺ z ( x ′ ) , normalized by the total number of trees |T | , gives the ratio of trees in the ensemble for which x and x ′ fall into the same leaf.

Original: We modify the kernel by adding a trainable signal variance σ 20 [34].
Revised: We modify the kernel by adding a trainable signal variance σ 20 [42].

Revised: Note that the tree ensemble defining the kernel and the kernel hyperparameters are trained separately.

Original: Davies and Ghahramani [10]prove that the tree kernel is a suitable GP prior.
Revised: Davies and Ghahramani [15] prove that the tree kernel is a suitable GP prior.

Original: The resulting (non-stationary and supervised) treekernel describes a prior over piece-wise constant functions when used in a GP.
Revised: The resulting (non-stationary and supervised) tree kernel describes a prior over piece-wise constant functions when used in a GP.

Original: 3.2 Posterior distribution
Revised: 3.2 Posterior distribution

Original: We approximate f as a Gaussian process with zero mean and kernel k Tree : f ( · ) ∼ GP (0 , k Tree ) .
Revised: We approximate f as a Gaussian process with zero mean and kernel k Tree : f ( · ) ∼ GP (0 , k Tree ) .

Original: Sincek Tree is a valid Mercer kernel, the mean M ( x ) and variance V ( x ) of the GP at x ∈ R n is [45]:
Revised: Since k Tree is a valid Mercer kernel, the mean M ( x ) and variance V ( x ) of the GP at x ∈ R n is [57]:

Original: [Equation]
Revised: [Equation]

Original: The Gram matrix K X , X ∈ R m × m has entries describing pairwise correlations computed based on thekernel function in Eq.
Revised: The Gram matrix K X , X ∈ R m × m has entries describing pairwise correlations computed based on the kernel function in Eq.

Original: The entries of vector K x , X ∈ R m contain correlations between the inputx and sampled data points, defined as [ k Tree ( x , x 1 ) , k Tree ( x , x 2 ) , . . .
Revised: The entries of vector K x , X ∈ R m contain correlations between the input x and sampled data points, defined as [ k Tree ( x , x 1 ) , k Tree ( x , x 2 ) , . . .

Original: , k Tree ( x , x m )] with vectors x ireferring to rows in data set X .
Revised: , k Tree ( x , x m )] with vectors x i referring to rows in data set X .

Original: Target values
Revised: Target values

Original: y ∈ R m are the corresponding observations for X .
Revised: y ∈ R m are the corresponding observations for X .

Original: Eq.
Revised: Eq.

Original: (2) describes the noise-free case of the GP mean and variance.
Revised: (2) describes the noise-free case of the GP mean and variance.

Original: To fit a GP based on the treekernel function, we usually require a noise term, i.e. a diagonal matrix σ 2 y I that is added to K X , X .
Revised: To fit a GP based on the tree kernel function, we usually require a noise term, i.e., a diagonal matrix σ 2 y I that is added to K X , X .

Original: We set hyperparameters σ 2 y and σ 2 0 by maximizing the log marginal likelihood.
Revised: We set hyperparameters σ 2 y and σ 20 by maximizing the log marginal likelihood.

Revised: Lee et al.

Revised: [42] compute the Eq.

Revised: (2a) inverse of the Gram matrix K X , X efficiently by exploiting the property that the rank of K X , X is at most the number of leaves over trees.

Original: Fig.
Revised: Fig.

Original: 1 visualizes the Eq.
Revised: 1 visualizes the Eq.

Original: (2) M ( x ) , V ( x ) and the lower-confidence bound (LCB) [9] response surface.
Revised: (2) M ( x ) , V ( x ) and the upper-confidence bound (UCB) [14] response surface.

Original: Fig.
Revised: Fig.

Original: 1(a) showsthat the tree kernel-based GP mean gives a good piecewise-constant approximation of  the Braninfunction.
Revised: 1(a) shows that the tree kernel-based GP mean gives a good piecewise-constant approximation of negative one times the Branin function.

Original: Fig. 1(b) shows that variance peaks reveal areas where data are sparse, reliably identifyinguncertainty in the underlying surrogate model.
Revised: Fig. 1(b) shows that variance peaks reveal areas where data are sparse, reliably identifying uncertainty in the underlying surrogate model.

Original: Finally, Fig.
Revised: Finally, Fig.

Original: 1(c) shows how an acquisition functionsuch as the LCB can effectively manage the exploitation-exploration trade-off. 
Revised: 1(c) shows how an acquisition function such as the UCB can effectively manage the exploitation-exploration trade-off. [Figure]

Original: 3.3 Global optimization of tree ensembles
Revised: 3.3 Global optimization of tree ensembles

Original: Miši´c [38] proposes a mixed-integer linear optimization formulation that ensures that binary variablesz t,l follow the logic of the tree ensemble:
Revised: Miši ´ c [49] proposes a mixed-integer linear optimization formulation that ensures that binary variables z t,l follow the logic of the tree ensemble:

Original: [Equation]
Revised: [Equation]

Original:  ∀ t ∈ T, ∀ s ∈ splits
Revised: l ∈L [Equation] ∀ t ∈ T

Original: [Equation]
Revised: [Equation]

Original: t
Revised: l

Original: [Equation]
Revised: [Equation]

Original: l ∈ right
Revised: ∀ t ∈ T , ∀ s ∈ splits ( t ) ,

Original: [Equation]
Revised: [Equation]

Revised: 3b)

Original: [Equation]
Revised: [Equation]

Original: ∀ t ∈ T , ∀ s ∈ splits ( t ) , (3c)
Revised: ∀ t ∈ T , ∀ s ∈ splits ( t ) [Equation] 3c [Equation]

Revised: ν

Original: [Equation]
Revised: [Equation]

Original: ∀ i ∈ C , (3d) 
Revised: ∀ i ∈ C , (3d) ν

Original: [Equation]
Revised: [Equation]

Original: ∀ i ∈ N , ∀ j ∈
Revised: ∀ i ∈ N , ∀ j ∈

Original: [ K i − 1] 
Revised: [ K i − 1] , (3e) ν i

Original: [Equation]
Revised: [Equation]

Original: 3e)
Revised: ∈ {

Original: [Equation]
Revised: [Equation]

Original: ∀ i ∈ [ n ] , ∀ j ∈
Revised: ∀ i ∈ [ n ] , ∀ j ∈

Original: [ K i ]

Original: [Equation] 3f) [Equation] ∀ t ∈ T , ∀ l ∈ L t .
Revised: [ K i ] , (3f) z t,l ≥ 0 , ∀ t ∈ T , ∀ l ∈ L t .

Original: (3g)
Revised: (3g)

Original: Eq. (3a) guarantees exactly one active leaf l in leaf set L t for tree t .
Revised: Eq. (3a) guarantees exactly one active leaf l in leaf set L t for tree t .

Original: Eqs.
Revised: Eqs.

Original: (3b)–(3c) ensure that binaryvariables z t,l are only active if all previous splits ν V ( s ) ,j are active.
Revised: (3b)–(3c) ensure that binary variables z t,l are only active if all previous split binaries

Revised: ν V ( s ) ,j corresponding to continuous splitting thresholds are active.

Original: At any node in a given tree, left ( s )and right ( s ) contain all leaves following the left and right branches, respectively.
Revised: At any node in a given tree, left ( s ) and right ( s ) contain all leaves following the left and right branches, respectively.

Original: The mapping V ( s )gives the splitting feature at node s ∈ splits ( t ) in tree t  .
Revised: The mapping V ( s ) gives the splitting feature at node s ∈ splits ( t ) in tree t , with splits ( t ) defining the set of all splits in tree t .

Original: Tree ensembles can handle continuous,integer and categorical data.
Revised: Tree ensembles can handle continuous, integer and categorical data.

Original: Continuous splits s are defined by x V ( s ) ≤ v V ( s ) ,j conditions, wherev V ( s ) ,j is the learned splitting threshold.
Revised: Continuous splits s are defined by x V ( s ) ≤ v V ( s ) ,j conditions, where v V ( s ) ,j is the learned splitting threshold.

Original: Therefore, C ( s ) only contains a single index j representingthe threshold v V ( s ) ,j .
Revised: Therefore, C ( s ) only contains a single index j representing the threshold v V ( s ) ,j .

Original: Categorical splits s are characterized by (subsets of) categories available forfeature V ( s ) and define a splitting condition based on the inclusion of x V ( s ) in the category subset atsplit s .
Revised: Categorical splits s are characterized by (subsets of) categories available for feature V ( s ) and define a splitting condition based on the inclusion of x V ( s ) in the category subset at split s .

Original: For categorical splits, C ( s ) includes the category indices comprising the category subset atsplit s .
Revised: For categorical splits, C ( s ) includes the category indices comprising the category subset at split s .

Original: Eq.
Revised: Eq.

Original: (3d) ensures that only one category is active per categorical variable i ∈ C .
Revised: (3d) ensures that only one category is active per categorical variable i ∈ C .

Original: Continuoussplitting thresholds of all trees in the ensemble are ordered according to v i, 1 < v i, 2 < ... <
Revised: Continuous splitting thresholds of all trees in the ensemble are ordered according to v i, 1 < v i, 2 < ... <

Original: v i,K iwith K i denoting the index for the last split of continuous feature i ∈ N .
Revised: v i,K i with K i denoting the index for the last split of continuous feature i ∈ N .

Original: To enforce this order, Eq.
Revised: To enforce this order, Eq.

Original: (3e) ensures that binary variables
Revised: (3e) ensures that binary variables

Original: ν
Revised: ν

Original: i,j , corresponding to the split thresholds v i,j , are activatedsequentially.
Revised: i,j , corresponding to the split thresholds v i,j , are activated sequentially.

Original: The model comprising Eqs.
Revised: The model comprising Eqs.

Original: (3) has no direct dependency on x and is fully definedby binary variables indicating which splits and leaves of the tree model active.
Revised: (3) has no direct dependency on x and is fully defined by binary variables indicating which splits and leaves of the tree model active.

Original: However, to allowthe user to include extra equality and inequality constraints on the input vector x , we bound thecontinuous variables based on the active splits by adding linking constraints [39]:
Revised: However, to allow the user to include extra equality and inequality constraints on the input vector x , we bound the continuous variables based on the active splits by adding linking constraints [50]: x i ≥ v Li + K i (cid:88) [Equation] ∀

Revised: i

Original: [Equation]
Revised: [Equation]

Revised: N

Revised: [Equation]

Revised: 4a

Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: = { j ∈
Revised: ∀ i [Equation] N

Original: [ K
Revised: [Equation] x [Equation] (cid:2)

Original: [Equation]
Revised: [Equation]

Original: ] |
Revised: (cid:3) , ∀ i ∈ N , ( [Equation] ) x

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Revised: 1 } , ∀ i ∈ C , (4d)

Original: For categorical variables, Eq.
Revised: For categorical variables, Eq.

Original: (4d) maps the indices of active categories onto x i for i ∈ C .
Revised: (4d) maps the indices of active categories onto x i for i ∈ C .

Original: Theseoptimization formulations are implemented in open-source software ENTMOOT [53] and OMLT [7].
Revised: These optimization formulations are implemented in open-source software ENTMOOT [66] and OMLT [11].

Original:  Tree ensemble kernels for Bayesian optimization
Revised:  Tree ensemble kernels for Bayesian optimization

Original: The technical details in Section 3 are insufficient to use the tree ensemble kernel in a Bayesianoptimization framework.
Revised: The technical details in Section 3 are insufficient to use the tree ensemble kernel in a Bayesian optimization framework.

Original: While Eqs.
Revised: While Eqs.

Original: 3 and 4 allow optimization over the mean of an associatedacquisition function [38], Bayesian optimization also requires quantifying model uncertainty forexploration.
Revised: 3 and 4 allow optimization over the mean of an associated acquisition function [49], Bayesian optimization also requires quantifying model uncertainty for exploration.

Original: This section proposes a mixed-integer second-order cone optimization formulationto capture the standard deviation of a GP with a tree kernel prior.
Revised: This section proposes a mixed-integer second-order cone optimization formulation to capture the standard deviation of a GP with a tree kernel prior.

Original: Combining this optimization formulation with the already-developed mixed-integer formulation of the mean function, we derivethe lower confidence bound (LCB) of the tree kernel-based GP.
Revised: Combining this optimization formulation with the already-developed mixed-integer formulation of the mean function, we derive the upper-confidence bound (UCB) of the tree kernel-based GP.

Original: The advantage of deriving a mixedinteger second-order cone optimization formulation of the LCB acquisition function is that we canglobally optimize the acquisition function.
Revised: The advantage of deriving a mixedinteger second-order cone optimization formulation of the UCB acquisition function is that we can globally optimize the acquisition function.

Original: Additionally, we can easily incorporate explicit inputconstraints that capture domain knowledge and/or known search space relationships.
Revised: Additionally, we can easily incorporate explicit input constraints that capture domain knowledge and/or known search space relationships.

Original: Our optimization problem, which includes the LCB acquisition function, is:
Revised: Our optimization problem, which includes the UCB acquisition function, is:

Original: [Equation]
Revised: [Equation]

Original: with µ ( x ) and σ ( x ) denoting the surrogate model’s mean prediction and standard deviation, respectively.
Revised: with µ ( x ) and σ ( x ) denoting the surrogate model’s mean prediction and standard deviation, respectively.

Original: Functions h
Revised: Functions h

Original: ( x ) and g ( x ) are known constraints and handled similarly to [5, 55].
Revised: ( x ) and g ( x ) are known constraints and handled similarly to [9, 69].

Original: In ourimplementation, the constraints can be linear, quadratic, or polynomial.
Revised: In our implementation, the constraints can be linear, quadratic, or polynomial.

Original: Hyperparameter κ controlsthe exploitation-exploration trade-off to determine the next black-box function query area.
Revised: Hyperparameter κ ≥controls the exploitation-exploration trade-off to determine the next black-box function query area.

Original: Thesolution to Eq.
Revised: The solution to Eq.

Original: 5a is defined by x ∗ lb and x ∗ ub for non-categorical variables i ∈ N , i.e. continuousand integer features, and x ∗ cat , a set of valid category subsets for categorical variables i ∈ C .
Revised: 5a is defined by x ∗ lb and x ∗ ub for non-categorical variables i ∈ N , i.e., continuous and integer features, and x ∗ cat , a set of valid category subsets for categorical variables i ∈ C .

Original: Twovectors ( x ∗ lb and x ∗ ub ) define the non-categorical variables because the trees are piecewise-constantover intervals and the vectors define the lower and upper bounds of these intervals.
Revised: Two vectors ( x ∗ lb and x ∗ ub ) define the non-categorical variables because the trees are piecewise-constant over intervals and the vectors define the lower and upper bounds of these intervals.

Original: Next, we formalize the mean and variance of the tree kernel-based GP.
Revised: Next, we formalize the mean and variance of the tree kernel-based GP.

Original: Eqs.
Revised: Eqs.

Original: 6 and 2 are equivalent.
Revised: 6 and 2 are equivalent.

Original: [Equation]
Revised: [Equation]

Original: The Gram matrix K X , X and target vector y are constants in the optimization model as these quantitiesonly depend on the data set X .
Revised: The Gram matrix K X , X and target vector y are constants in the optimization model, as these quantities only depend on the data set X .

Original: The value K x , x is directly related to signal variance hyperparameterσ 0 since there is full leaf overlap for two identical inputs:
Revised: The value K x , x is directly related to signal variance hyperparameter σ 0 since there is full leaf overlap for two identical inputs:

Original: [Equation]
Revised: [Equation]

Original: The vector K x , X describes the kernel output of x with individual data points x i .
Revised: The vector K x , X contains the kernel output of x with individual data points x i .

Original: We computethe constant matrix A ∈ R m ×|L| after the gradient-boosted tree is trained but before solving theacquisition function.
Revised: We compute the constant matrix A ∈ R m ×|L| after the gradient-boosted tree is trained but before solving the acquisition function.

Original: The entries of A are equal to 1 for all active leaves l of data point i and 0otherwise.
Revised: The entries of A are equal to 1 for all active leaves l of data point i andotherwise.

Original: Given matrix A , Eqs.
Revised: Given matrix A , Eqs.

Original: (8) capture the kernel output by summing up all binary variables z t,lthat are active for data point i . 
Revised: (8) capture the kernel output by summing over binary variables z t,l that are active for data point i . [Equation] [Equation] l ∈L t

Revised: A

Original: [Equation]
Revised: [Equation]

Revised: ∀

Revised: i

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: 8b)

Original: To get an intuition for Eqs.
Revised: To get an intuition for Eqs.

Original: (8), note that the largest possible value for each k Tree ( x , x i ) is σ 20 (if thecorresponding entries of matrix
Revised: (8), note that the largest possible value for each k Tree ( x , x i ) is σ 20 (if the corresponding entries of matrix

Original: A are all equal to 1 and there is full leaf overlap) and the smallestpossible value for each k Tree ( x , x i ) is 0 (if there is no leaf overlap).
Revised: A are all equal to 1, and there is full leaf overlap) and the smallest possible value for each k Tree ( x , x i ) is 0 (if there is no leaf overlap).

Original: In general, values of each elementof vector K x , X will range between 0 and σ 20 : higher values in the elements of K x , X indicate a higherdegree of overlap between the next query location x and data point x i in the set X .
Revised: In general, values of each element of vector K x , X will range between 0 and σ 20 : higher values in the elements of K x , X indicate a higher degree of overlap between the next query location x and data point x i in the set X .

Original: Without considering additional tree model constraints h and g , the resulting acquisition function is amixed-integer quadratic optimization problem.
Revised: Without considering additional tree model constraints h and g , the resulting acquisition function is a mixed-integer quadratic optimization problem.

Original: The optimization problem is mixed-integer becauseof binary variables ν and quadratic because of Eq.
Revised: The optimization problem is mixed-integer because of binary variables ν and quadratic because of Eq.

Original: (6b).
Revised: (6b).

Original: The quadratic Eq.
Revised: The quadratic Eq.

Original: (6b) components areσ 2 and the terms k 2 Tree ( x , x i ) arising from the inner product of K x , X with itself.
Revised: (6b) components are σ 2 and the terms k 2 Tree ( x , x i ) arising from the inner product of K x , X with itself.

Original: We only requireone direction of the Eq.
Revised: We only require one direction of the Eq.

Original: (6b) equality ( ≤ ) and re-write Eq.
Revised: (6b) equality ( ≤ ) and re-write Eq.

Original: (6b) as a second-order cone constraint.
Revised: (6b) as a second-order cone constraint.

Revised: Second-order cone programming [1, 41, 45] optimizes over a linear objective subject to both linear and second-order cone constraints (here, convex quadratic constraints, but the theory is more general).

Revised: More recently, solvers integrate advanced methods solving second-order cone problems in the mixedinteger setting [4, 20, 46].

Original: Solver Gurobi 9 [21] automatically finds that Eq.
Revised: Solver Gurobi 9 [28] automatically finds that Eq.

Original: (6b) (with ≤ rather than = ) can be represented as asecond-order cone and makes the appropriate algorithm modifications, e.g. as described by [26, 57].
Revised: (6b) (with ≤ rather than = ) can be represented as a second-order cone and makes the appropriate algorithm modifications, e.g., as described by [33, 71].

Revised: The proposed formulation is also compatible with open-source solver alternatives including Bonmin [8], MindtPy [6], Pajarito [13], and SHOT [47].

Original: The acquisition function, with Objective 5a and Constraints (3)–(8b), is a mixed-integer second-ordercone program which can be solved with optimization solvers.
Revised: The acquisition function, with Objective 5a and Constraints (3)–(8b), is a mixed-integer second-order cone program which can be solved with optimization solvers.

Original: A valid solution to the proposed modelis a set of active leaves L ∗ and the intersection of all leaf subspaces [ x ∗ lb , x ∗ ub ] , x ∗ cat .
Revised: A valid solution to the proposed model is a set of active leaves L ∗ and the intersection of all leaf subspaces [ x ∗ lb , x ∗ ub ] , x ∗ cat .

Original: To derive the next black-box function query point x ∗ , we propose some heuristics.
Revised: To derive the next black-box function query point x ∗ , we propose some heuristics.

Original: The acquisitionfunction value is constant for x ∈ [ x ∗ lb , x ∗ ub ] , x ∗ cat , i.e. all contained points are equivalent from the perspective of the tree kernel-GP.
Revised: The acquisition function value is constant for x ∈ [ x ∗ lb , x ∗ ub ] , x ∗ cat , i.e., all contained points are equivalent from the perspective of the tree kernel-GP.

Original: For the continuous and integer features, we observe that tree modelstend to learn split thresholds close to training data points.
Revised: For the continuous and integer features, we observe that tree models tend to learn split thresholds close to training data points.

Original: We propose the center of [ x ∗ lb , x ∗ ub ] as thenext query point x ∗ for continuous and integer features:
Revised: We propose the center of [ x ∗ lb , x ∗ ub ] as the next query point x ∗ for continuous and integer features:

Original: [Equation]
Revised: [Equation]

Original: For integer features with a fractional mid-point between the upper and lower bound, we randomlyselect its floor or ceiling.
Revised: For integer features with a fractional mid-point between the upper and lower bound, we randomly select its floor or ceiling.

Original: For categorical features, we sample from the subset of available categories:
Revised: For categorical features, we sample from the subset of available categories:

Original: [Equation]
Revised: [Equation]

Original: For the unconstrained case, we use x ∗ mid as the next query point x ∗ .
Revised: For the unconstrained case, we use x ∗ mid as the next query point x ∗ .

Original: When additional input constraintsh ( x ) and/or g ( x ) are given, we know that at least one point in [ x ∗ lb , x ∗ ub ] , x ∗ cat is feasible even if theheuristic x ∗ mid is infeasible.
Revised: When additional input constraints h ( x ) and/or g ( x ) are given, we know that at least one point in [ x ∗ lb , x ∗ ub ] , x ∗ cat is feasible even if the heuristic x ∗ mid is infeasible.

Original: To repair the solution x ∗ mid , we project it onto the feasible space:x ∗ ∈ arg min
Revised: To repair the solution x ∗ mid , we project it onto the feasible space: [Equation] 11a

Original: [Equation]
Revised: [Equation]

Original: g
Revised: 11b

Original: [Equation]
Revised: [Equation]

Revised: 11c

Original: [Equation]
Revised: [Equation]

Original: Eq.
Revised: Eq.

Original: (11) projects x ∗ mid onto the feasible set defined by h ( x ) and g ( x ) .
Revised: (11) projects x ∗ mid onto the feasible set defined by h ( x ) and g ( x ) .

Original: The time complexity ofsolving Eq. 11, and that of solving Eqs.
Revised: The time complexity of solving Eq. 11, and that of solving Eqs.

Original: (3)–(6) for the search space [ x ∗ lb , x ∗ ub ] , x ∗ cat , are both NPhard.
Revised: (3)–(6) for the search space [ x ∗ lb , x ∗ ub ] , x ∗ cat , are both NP-hard.

Original: However,  we found that Gurobi 9 often solves both problems to ϵ -global optimality formoderately-sized tree models in less time compared to random sampling.
Revised: However, in preliminary evaluations we found that Gurobi 9 often solves both problems to ϵ -global optimality for moderately-sized tree models in less time compared to random sampling.

Original: Hyperparameters.
Revised: Hyperparameters.

Original: We train the kernel hyperparameters signal variance σ 20 and noise σ 2 y by maximizing the log marginal likelihood.
Revised: We train the kernel hyperparameters signal variance σ 20 and noise σ 2 y by maximizing the log marginal likelihood.

Original: Additional hyperparameters are introduced by the gradient-boostedtree ensemble, i.e. maximum tree depth and number of trees, and through κ in the LCB.
Revised: Additional hyperparameters are introduced by the gradient-boosted tree ensemble trained at every iteration, i.e., maximum tree depth and number of trees, and through κ in the UCB.

Original: For the Section 5 numerical studies, we leave κ and gradient-boosting hyperparameters constant and only increase maximum tree depth and number of trees for the high-dimensional CIFAR-NAS benchmark tocapture more complicated interactions.
Revised: For the Section 5 numerical studies, we leave κ and gradient-boosting hyperparameters constant and only increase maximum tree depth and number of trees for the high-dimensional CIFAR-NAS benchmark to capture more complicated interactions.

Original: The appendix reports specific values for all hyperparameters.
Revised: The appendix reports specific values for all hyperparameters.

Original: Limitations.
Revised: Limitations.

Original: The method suffers from standard BO challenges where the tree kernel may not bea good prior for the underlying black-box function, e.g. purely continuous feature spaces.
Revised: The method suffers from standard BO challenges where the tree kernel may not be a good prior for the underlying black-box function, e.g., purely continuous feature spaces.

Original: Morelimitations arise from solving an NP-hard problem to global optimality when working with large datasets in high-dimensional search spaces.
Revised: More limitations arise from solving an NP-hard problem to global optimality when working with large data sets in high-dimensional search spaces.

Original: For cases where solving the NP-hard problem is too difficultbut BO is still applicable, Gurobi 9 can typically develop good feasible solutions as a heuristic.
Revised: For cases where solving the NP-hard problem is too difficult but BO is still applicable, Gurobi 9 can typically develop good feasible solutions as a heuristic.

Original:  Numerical studies
Revised:  Numerical studies

Original: This section empirically evaluates the performance of tree kernel-based GPs using a wide varietyof synthetic and real-world benchmark problems.
Revised: This section empirically evaluates the performance of tree kernel-based GPs using a wide variety of synthetic and real-world benchmark problems.

Original: We show (i) the ability of tree kernels to captureuncertainty of the underlying tree ensemble, (ii) the advantage of using global vs. local strategies foroptimizing the acquisition function and (iii) the proposed algorithm’s superior performance in caseswith constrained search spaces and mixed variable types.
Revised: We show (i) the ability of tree kernels to capture uncertainty of the underlying tree ensemble, (ii) the advantage of using global vs. local strategies for optimizing the acquisition function and (iii) the proposed algorithm’s superior performance in cases with constrained search spaces and mixed variable types.

Original: LEAF-GP denotes the Section 4 proposedalgorithm, and Section 2 outlines the baseline of methods we compare against.
Revised: LEAF-GP denotes the Section 4 proposed algorithm, and Section 2 outlines the baseline of methods we compare against.

Original: For every run, wevisualize the median and confidence intervals of the first and third quartile based on 20 individualruns with varying random seeds.
Revised: For every run, we visualize the median and confidence intervals of the first and third quartile based on 20 individual runs with varying random seeds.

Original: Further technical details can be found in the appendix.
Revised: Further technical details can be found in the appendix.

Original: 5.1 Uncertainty metric
Revised: 5.1 Uncertainty metric

Revised: [Figure]

Original: The tree kernel-based GP uses the leaf overlap measure to quantify correlation between two inputs.
Revised: The tree kernel-based GP uses the leaf overlap measure to quantify correlation between two inputs.

Original: In BO, such measures help identify unexplored areas where correlation to existing training data islow and we expect inaccurate model predictions.
Revised: In BO, such measures help identify unexplored areas where correlation to existing training data is low and we expect inaccurate model predictions.

Original: To empirically test the tree kernel’s capability ofidentifying uncertainty in the underlying tree ensemble, we change the optimization formulation:
Revised: To empirically test the tree kernel’s capability of identifying uncertainty in the underlying tree ensemble, we change the optimization formulation:

Original: [Equation]

Original: , x ∗ cat ∈ arg min x , z ,ν µ ( x ) ,

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: 12b)

Original: [Equation]
Revised: [Equation]

Original: 0.9 Tree Agreement Ratio

Original: R e l a t i v e

Original: P r e d i c t i o n

Original: E rr o r (a) Relative Model Error 0.

Original: 0.9 Tree Agreement Ratio 2.

Original: P r e d i c t i o n M e a n [Figure] Eq. (12a) minimizes over the mean prediction of the tree kernel-based GP with Eq.
Revised: Eq. (12a) maximizes over the mean prediction of the tree kernel-based GP with Eq.

Original: (12c) restrictingthe ratio of tree agreement for the black-box query point x ∗ defined as the Eq.
Revised: (12c) restricting the ratio of tree agreement for the black-box query point x ∗ defined as the Eq.

Original: (9) center of theoptimal area x ∗ lb , x ∗ ub , x ∗ cat .
Revised: (9) center of the optimal area x ∗ lb , x ∗ ub , x ∗ cat .

Original: A solution to problem Eq.
Revised: A solution to problem Eq.

Original: (12) guarantees a maximum leaf overlap of
Revised: (12) guarantees a maximum leaf overlap of

Original: R between x ∗ and all available data points X .
Revised: R between x ∗ and all available data points X .

Revised: Limiting the tree agreement ratio corresponds to constraining the Eq.

Revised: (1) kernel correlation between the existing dataset and the proposed optimal area x ∗ lb , x ∗ ub , x ∗ cat .

Original: Here, we evaluate values of R ∈ [0 . 35 , 1 . 0] withincrements of 0 . 05 and compute the model error according to:
Revised: Here, we evaluate values of R ∈ [0 . 35 , 1 . 0] with increments of 0 . 05 and compute the model error according to:

Original: [Equation]
Revised: [Equation]

Original: Fig.
Revised: Fig.

Original: 2 shows results for the Rastrigin [51] benchmark.
Revised: 2 shows results for the Rastrigin [63] benchmark.

Original: With smaller values of R , the Eq.
Revised: With smaller values of R , the Eq.

Original: (12)optimization problem becomes more restrictive, leading to larger objective values for solution x ∗ .
Revised: (12) optimization problem becomes more restricted, leading to smaller objective values for solution x ∗ .

Original: Fig. 2(a) shows that a growing leaf overlap, i.e. larger R , reduces the model error defined in Eq.
Revised: Fig. 2(a) shows that a growing leaf overlap (increasing R ) reduces the model error defined in Eq.

Original: This suggests that the kernel works as intended and that low kernel correlation can reveal searchspace areas with high model uncertainty.
Revised: This suggests that the kernel works as intended and that low kernel correlation can reveal search space areas with high model uncertainty.

Original: The appendix provides results for additional benchmarks.
Revised: The appendix provides results for additional benchmarks.

Original: 5.2 Local vs. global acquisition optimization
Revised: 5.2 Local vs. global acquisition optimization

Original: [Figure]
Revised: [Figure]

Revised: o x

Revised: [Figure]

Revised: j

Revised: [Figure]

Original:  This section (i) compares LEAF-GP to other state-of-the-art algorithms for common benchmarks and(ii) shows the advantage of global vs. local strategies for optimizing the acquisition function.
Revised: c t i v [Figure] This section (i) compares LEAF-GP to other state-of-the-art algorithms for common benchmarks and (ii) shows the advantage of global vs. local strategies for optimizing the acquisition function.

Original: Weintroduce a variation of the proposed algorithm, LEAF-GP-RND , which optimizes the same acquisitionfunction as LEAF-GP , but uses random sampling instead of Gurobi 9.
Revised: We introduce a variation of the proposed algorithm, LEAF-GP-RND , which optimizes the same acquisition function as LEAF-GP , but uses random sampling instead of Gurobi 9.

Original: Fig.
Revised: Fig.

Original: 3 shows results for
Revised: 3 shows results for

Original: Hartmann (6D) [51] and Styblinski-Tang (10D) [51].
Revised: Hartmann (6D) [63] and Styblinski-Tang (10D) [63].

Original: We do not expect tree model-based approachesto perform well since both benchmark functions are continuous.
Revised: We do not expect tree model-based approaches to perform well since both benchmark functions are continuous.

Original: However, the benchmarks show thatall approaches perform similarly for Hartmann (6D) with UCB-MATERN and EI-MATERN improving theblack-box objective at the fastest rate.
Revised: However, the benchmarks show that all approaches perform similarly for Hartmann (6D) with UCB-MATERN and EI-MATERN improving the black-box objective at the fastest rate.

Original: For Styblinski-Tang (10D), LEAF-GP significantly outperformsother algorithms.
Revised: For Styblinski-Tang (10D), LEAF-GP significantly outperforms other algorithms.

Original: Although LEAF-GP is not specialized to this setting, observe that it performsroughly with the state-of-the-art on these continuous, unconstrained optimization problems.
Revised: Although LEAF-GP is not specialized to this setting, observe that it performs roughly equivalently to the state of the art on these continuous, unconstrained optimization problems.

Original: Although sampling-based LEAF-GP-RND and optimization-based LEAF-GP perform similarly on thesmaller Hartmann (6D), LEAF-GP is particularly strong on the higher-dimensional Styblinski-Tang(10D) benchmark function, and two additional benchmarks in the appendix.
Revised: Although sampling-based LEAF-GP-RND and optimization-based LEAF-GP perform similarly on the smaller Hartmann (6D), LEAF-GP is particularly strong on the higher-dimensional Styblinski-Tang (10D) benchmark function, and two additional benchmarks in the appendix.

Original: Moreover, LEAF-GP-RNDdoes not support explicit input constraints and performs particularly bad for BO with known con- straints.
Revised: Moreover, LEAF-GP-RND does not support explicit input constraints and performs particularly bad for BO with known constraints.

Original: For the following constrained benchmarks, we remove LEAF-GP-RND from the comparison.
Revised: For the following constrained benchmarks, we remove LEAF-GP-RND from the comparison.

Original: The appendix has additional details and results.
Revised: The appendix has additional details and results.

Original: [Figure]

Original:  Constrained spaces
Revised: 5.3 Constrained spaces

Original: [Figure]

Original: median line and confidence intervals (first and third quartile) from 20 random seeds.

Original: Confidence intervals are neglected for methods that cannot improve the initial training data.

Original: Figure subtitles give the function name and number of: dimensions (D), equality constraints (EC), and inequality constraints (IC).

Original: Section 5.3 provides more details.

Original: This section presents numerical benchmarks with known input constraints.
Revised: This section presents numerical benchmarks with known input constraints.

Original: The acquisition optimization strategy of LEAF-GP allows explicit consideration of input constraints, i.e. logical andconvex/non-convex n -th degree polynomial equality and inequality constraints for mixed variablespaces.
Revised: The acquisition optimization strategy of LEAF-GP allows explicit consideration of input constraints, i.e., logical and convex/non-convex n -th degree polynomial equality and inequality constraints for mixed variable spaces.

Original: The UCB-MATERN and EI-MATERN implementations of BoTorch supports only linear equalityand inequality constraints at the acquisition optimization step.
Revised: The UCB-MATERN and EI-MATERN implementations of BoTorch supports only linear equality and inequality constraints at the acquisition optimization step.

Original: The GA pymoo algorithm has aninterface for callable constraint functions which are considered when generating new generations ofcandidate points.
Revised: The GA pymoo algorithm has an interface for callable constraint functions which are considered when generating new generations of candidate points.

Original: When a method does not support the specific input constraints, we penalize theobjective:
Revised: When a method does not support the specific input constraints, we penalize the objective:

Original: [Equation]
Revised: [Equation]

Original:  where g ( x ) ≤ 0 and h ( x ) = 0 are inequality and equality constraints, respectively.
Revised: (14) where g ( x ) ≤ 0 and h ( x ) = 0 are inequality and equality constraints, respectively.

Original: This penaltystrategy allows methods that do not support explicit constraints to still produce feasible points givenenough iterations.
Revised: This penalty strategy allows methods that do not support explicit constraints to still produce feasible points given enough iterations.

Original: Eq.
Revised: Eq.

Original: (14) introduces the hyperparameter λ which weights the penalty, we test valuesλ ∈ { 1 , 10 , 100 } and only plot the best run for all methods that rely on constraint penalization.
Revised: (14) introduces the hyperparameter λ which weights the penalty, we test values λ ∈ { 1 , 10 , 100 } and only plot the best run for all methods that rely on constraint penalization.

Original: Toinitialize every method with feasible points, we draw random samples from a uniform distributionwithin the bounds that define the search space and compute the closest feasible point similar to
Revised: To initialize every method with feasible points, we draw random samples from a uniform distribution within the bounds that define the search space and compute the closest feasible point similar to

Original: Eq.
Revised: Eq.

Original: For constraint benchmarks, we introduce FEAS-RANDOM which simply projects randomsamples onto the set of feasible points that satisfy h ( x ) and g ( x ) .
Revised: For constrained benchmarks, we introduce FEAS-RANDOM which simply projects random [Figure] function name and number of: dimensions (D), equality constraints (EC), and inequality constraints (IC).

Revised: Section 5.3 provides more details.

Revised: samples onto the set of feasible points that satisfy h ( x ) and g ( x ) .

Original: Fig.
Revised: Fig.

Original: 4 plots feasible solutionsto four different continuous benchmark problems, G1, G3, G4 [25], and Alkylation [48].
Revised: 4 plots feasible solutions to four different continuous benchmark problems, G1, G3, G4 [32], and Alkylation [60].

Original: The G1benchmark has linear inequality constraints only, which are supported by UCB-MATERN , EI-MATERNand LEAF-GP .
Revised: The G benchmark has linear inequality constraints only, which are supported by UCB-MATERN , EI-MATERN and LEAF-GP .

Original: Fig.
Revised: Fig.

Original: 4(a) shows LEAF-GP making quick progress at the beginning with UCB-MATERNcatching up towards the end.
Revised: 4(a) shows LEAF-GP making quick progress at the beginning with UCB-MATERN catching up towards the end.

Original: G3 and G4 have different combinations of nonlinear constraints,and LEAF-GP significantly outperforms competing methods and random feasible sampling.
Revised: G3 and G4 have different combinations of nonlinear constraints, and LEAF-GP significantly outperforms competing methods and random feasible sampling.

Original: The
Revised: The

Original: Alkylation benchmark determines optimal operating conditions for a simplified alkylation processconsidering complicated nonlinear constraints representing economic, physical and performancelimits.
Revised: Alkylation benchmark determines optimal operating conditions for a simplified alkylation process considering complicated nonlinear constraints representing economic, physical and performance limits.

Original: Again, LEAF-GP outperforms other methods, which often struggle to find feasible solutions.
Revised: Again, LEAF-GP outperforms other methods, which often struggle to find feasible solutions.

Original: More details regarding the presented benchmark problems are given in the appendix.
Revised: More details regarding the presented benchmark problems are given in the appendix.

Original: [Figure]
Revised: 5.4 Mixed-variable spaces

Original: We now consider spaces that exhibit mixtures of continuous, integer and categorical variables forconstrained problems.
Revised: We now consider spaces that exhibit mixtures of continuous, integer and categorical variables for constrained problems.

Original: Tree model-based algorithms naturally support categorical variables byreplacing continuous splits with categorical splits.
Revised: Tree model-based algorithms naturally support categorical variables by replacing continuous splits with categorical splits.

Original: For all methods that do not support categoricalfeatures, we use one-hot encoding.
Revised: For all methods that do not support categorical features, we use one-hot encoding.

Original: The category with the highest corresponding auxiliary variablevalue is chosen for the subsequent query point.
Revised: The category with the highest corresponding auxiliary variable value is chosen for the subsequent query point.

Original: On the Pressure Vessel (4D) [8] benchmark, whichcomprises two continuous features, two integer features and three inequality constraints, Fig. 5(a)shows that LEAF-GP outperforms other black-box optimization methods.
Revised: On the Pressure Vessel (4D) [12] benchmark, which comprises two continuous features, two integer features and three inequality constraints, Fig. 5(a) shows that LEAF-GP outperforms other black-box optimization methods.

Original: The CIFAR-NAS (29D) is a high-dimensional benchmark problem with one continuous, 23 integerand five categorical variables.
Revised: The CIFAR-NAS (29D) is a high-dimensional benchmark problem with one continuous, 23 integer and five categorical variables.

Original: It describes properties of different layers and training hyperparametersfor a convolutional neural network (CNN) trained on CIFAR-10 [33].
Revised: It describes properties of different layers and training hyperparameters for a convolutional neural network (CNN) trained on CIFAR-10 [40].

Original: The problem’s search spaceis hierarchical, as different layers, i.e. convolutional and fully-connected layers, can be activated.
Revised: The problem’s search space is hierarchical, as different layers, i.e., convolutional and fully-connected layers, can be activated.

Revised: [Figure]

Original: Properties describing a layer are only relevant if the layer is active.
Revised: Properties describing a layer are only relevant if the layer is active.

Original: For this problem, we introducetwo types of constraints to guide the LEAF-GP : (i) constraints that allow for feasible network designs,i.e. different values for padding, stride, kernel size and max pooling in earlier layers affects whatkernel size is feasible in following layers and (ii) constraints that ensure that hierarchies are respected.
Revised: For this problem, we introduce two types of constraints to guide the LEAF-GP : (i) constraints that allow for feasible network designs, i [Figure] e., different values for padding, stride, kernel size and max pooling in earlier layers affects what kernel size is feasible in following layers and (ii) constraints that ensure that hierarchies are respected.

Original: Towards the former, we include constraints to ensure the output size of every convolutional layer ispositive.
Revised: Towards the former, we include constraints to ensure the output size of every convolutional layer is positive.

Original: In general, tree models are particularly good at capturing hierarchical search space structures.
Revised: In general, tree models are particularly good at capturing hierarchical search space structures.

Original: To enforce hierarchical relationships during the acquisition optimization step, we introduce indicatorconstraints that force layer properties to take predefined default values if the layer is inactive.
Revised: To enforce hierarchical relationships during the acquisition optimization step, we introduce indicator constraints that force layer properties to take predefined default values if the layer is inactive.

Original: Thisensures that the optimizer avoids leaves that infer splitting conditions from feature properties ofinactive layers, thus effectively reducing the search space size.
Revised: This ensures that the optimizer avoids leaves that infer splitting conditions from feature properties of inactive layers, thus effectively reducing the search space.

Original: To expedite tests, we train the CNNon half of the CIFAR-10 training data and optimize for test accuracy.
Revised: To expedite tests, we train the CNN on half of the CIFAR-10 training data and optimize for test accuracy.

Original: Fig.
Revised: Fig.

Original: 5(b) summarizes theresults of this study and shows that tree model-based algorithms generally outperform UCB-MATERNand EI-MATERN .
Revised: 5(b) summarizes the results of this study and shows that tree model-based algorithms generally outperform UCB-MATERN and EI-MATERN .

Original: Utilizing search space constraints, SMAC and LEAF-GP find the neural architectureswith the highest test accuracy with SMAC slightly outperforming LEAF-GP .
Revised: Utilizing search space constraints, SMAC and LEAF-GP find the neural architectures with the highest test accuracy with SMAC slightly outperforming LEAF-GP .

Original: Since finding feasiblearchitectures for CIFAR-10 is not particularly challenging, Fig.
Revised: Since finding feasible architectures for CIFAR-10 is not particularly challenging, Fig.

Original: 9  shows a benchmark for tuningvariational autoencoders (VAE) adapted from Daxberger et al.
Revised: 9 in the appendix shows a benchmark for tuning variational autoencoders (VAE) adapted from Daxberger et al.

Original: [11] where LEAF-GP significantlyoutperforms other algorithms.
Revised: [16], where LEAF-GP significantly outperforms other algorithms.

Original: Finding feasible VAE architectures is exceptionally difficult given thatonly certain hyperparameter combinations decode the latent encoding  back tothe original  image size.
Revised: Finding feasible VAE architectures is more difficult given the requirement that the latent encoding must be decoded back to original input image size.

Original:  Conclusion
Revised:  Conclusion

Original: We present a framework for black-box optimization based on tree kernel Gaussian processes thatsimultaneously allows (i) reliable uncertainty quantification of mixed feature spaces and (ii) incorporation of explicit input constraints.
Revised: We present a framework for black-box optimization based on tree kernel Gaussian processes that simultaneously allows (i) reliable uncertainty quantification of mixed feature spaces and (ii) incorporation of explicit input constraints.

Original: Although these two needs have been considered separately, weare able to address both simultaneously through the mixed-integer second-order cone formulation of the acquisition function.
Revised: Although these two needs have been considered separately, we are able to address both simultaneously through the mixed-integer second-order cone formulation of the acquisition function.

Original: The numerical studies show that the proposed strategy performs competitively with state-of-the-art algorithms for unconstrained problems and may significantly outperform existing methods for constrained benchmarks, especially those with mixed feature spaces.
Revised: The numerical studies show that the proposed strategy performs competitively with state-of-the-art algorithms for unconstrained problems and may significantly outperform existing methods for constrained benchmarks, especially those with mixed feature spaces.

Original: We useoptimization constraints together with the acquisition functions to incorporate domain knowledge and leverage hierarchical search space structures, e g. for neural architecture search.Checklist
Revised: We use optimization constraints together with the acquisition functions to incorporate domain knowledge and leverage hierarchical search space structures, e.g., for neural architecture search.

Original: For all authors...

Original:  (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? [Yes] See Abstract and Section 1. (b) Did you describe the limitations of your work?
Revised:  Acknowledgements

Original: [Yes] See Section 4.

Original: (c) Did you discuss any potential negative societal impacts of your work?

Original: [No] There is

Original: no obvious societal impact of the method proposed.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?

Original: [Yes]

Original: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]

Original: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]

Original: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?

Original: [Yes] We submit the

Original: code torun all experiments for the proposed method. (b) Did you specify all the training details (e g., data splits, hyperparameters, how they
Revised: This work was supported by BASF SE, Ludwigshafen am Rhein, EPSRC Research Fellowships to

Original: were chosen)?

Original: [Yes] See end of Section 4 and supplemental material.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experi-

Original: ments multiple times)?

Original: [Yes] We run 20 random seeds for every experiment and report

Original: median and confidence intervals, i e. first and third quartile. See Section 5.
Revised: R.M. (EP/P016871/1) and CT (EP/T001577/1), and an Imperial College Research Fellowship to CT.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g.,

Original: type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] We report the computational

Original: resources used to conduct the experiments in the supplementary material.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] Yes, we cite alllibraries that were used when preparing the numerical studies.

Original: For more information,we refer to the supplemental material.

Original: (b) Did you mention the license of the assets?

Original: [No] License information for all assets canbe found in the documentation of each of the assets.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No]The code to reproduce the paper’s main results will be published open-source after thepeer review process.

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?

Original: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?

Original: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...

Original: [Table]

Original: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?

Original: [N/A]

Original: [Table]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: [Table]

Original: Board (IRB) approvals, if applicable?

Original: [N/A]

Original: [Table]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?

Original: [N/A]

Original: A AppendixB General experimental setup
Revised: A Appendix B General experimental setup

Original: All experimental results presented in Section 5 were evaluated on an HTCondor cluster (see [52]) ofmachines equipped with Intel Core i7-8700 3.20GHz and 16 GB RAM.
Revised: All experimental results presented in Section 5 were evaluated on an HTCondor cluster (see [64]) of machines equipped with Intel Core i7-8700 3.20GHz and 16 GB RAM.

Original: Confidence intervals showthe first and third quartile of 20 independent runs with random seeds ∈ [101 , 102 , . .
Revised: Confidence intervals show the first and third quartile of 20 independent runs with random seeds ∈ [101 , 102 , . .

Original: If notspecifically indicated in Section C, all competing algorithms use default values for all hyperparameters.
Revised: If not specifically indicated in Section C, all competing algorithms use default values for all hyperparameters.

Original: To allow for a fair comparison, we give the same set of initial data points to all tested methods forboth constrained and unconstrained benchmark problems.
Revised: To allow for a fair comparison, we give the same set of initial data points to all tested methods for both constrained and unconstrained benchmark problems.

Original: A set of ten initial samples is used for the
Revised: A set of ten initial samples is used for the

Original: Section 5.4 CIFAR-NAS example due to its high dimensionality.
Revised: Section 5.4 CIFAR-NAS example due to its high dimensionality.

Original: Five initial points are used for theremaining benchmark problems.
Revised: Five initial points are used for the remaining benchmark problems.

Original: C Algorithms
Revised: C Algorithms

Original: This section summarizes the different algorithms used for the Section 5 numerical studies.
Revised: This section summarizes the different algorithms used for the Section 5 numerical studies.

Original: We giveimplementation details and hyperparameter settings to reproduce the presented results.
Revised: We give implementation details and hyperparameter settings to reproduce the presented results.

Original: C.1 LEAF-GP and LEAF-GP-RND
Revised: C.1 LEAF-GP and LEAF-GP-RND

Original: The LEAF-GP uses LightGBM [30] for training gradient-boosted tree ensembles.
Revised: The LEAF-GP uses LightGBM [37] for training gradient-boosted tree ensembles.

Original: All runs use thehyperparameter value min_data_in_leaf = 1 as the training dataset size needs to be at least twice theminimum number of data points a leaf is based on.
Revised: All runs use the hyperparameter value min_data_in_leaf = 1 as the training dataset size needs to be at least twice the minimum number of data points a leaf is based on.

Original: The min_data_in_leaf default value of LightGBMis 20, which would cause run-time errors after initialization.
Revised: The min_data_in_leaf default value of LightGBM is 20, which would cause run-time errors after initialization.

Original: We also set LightGBM hyperparametermin_data_per_group = 1 .
Revised: We also set LightGBM hyperparameter min_data_per_group = 1 .

Original: For the high-dimensional benchmark problem CIFAR-NAS, we setmax_depth = 5 and num_boost_rounds = 100 for training the ensemble in LightGBM, referring tomaximum interaction depth per decision tree and total number of trees in the ensemble, respectively.
Revised: For the high-dimensional benchmark problem CIFAR-NAS, we set max_depth = 5 and num_boost_rounds = 100 for training the ensemble in LightGBM, referring to maximum interaction depth per decision tree and total number of trees in the ensemble, respectively.

Original: For all other benchmarks we use max_depth = 3 and num_boost_rounds = 50 .
Revised: For all other benchmarks we use max_depth = 3 and num_boost_rounds = 50 .

Original: We implement the tree ensemble kernel as a non-stationary kernel in GPyTorch [18].
Revised: We implement the tree ensemble kernel as a non-stationary kernel in GPyTorch [24].

Original: For derivingthe posterior distribution, we use a Gaussian likelihood and standardize the target values of the dataset.
Revised: For deriving the posterior distribution, we use a Gaussian likelihood and standardize the target values of the data set.

Original: Section 3 introduces signal variance σ 0 and noise term σ y as kernel hyperparameters which arefitted by maximizing the marginal log likelihood over 200 epochs using the Adam solver [31].
Revised: Section 3 introduces signal variance σ 0 and noise term σ y as kernel hyperparameters which are fitted by maximizing the marginal log likelihood over 200 epochs using the Adam solver [38].

Original: Thehyperparameters are constrained by intervals according to σ 0 ∈ [5e − 4 , 0 . 2] and σ y ∈ [0 . 05 , 20 . 0] .
Revised: The hyperparameters are constrained by intervals according to σ 0 ∈ [5e − 4 , 0 . 2] and σ y ∈ [0 . 05 , 20 . 0] .

Original: For LEAF-GP the Section 4 acquisition optimization formulation is encoded using gurobipy [21]and solved using Gurobi 9.
Revised: For LEAF-GP the Section 4 acquisition optimization formulation is encoded using gurobipy [28] and solved using Gurobi 9.

Original: Runs are limited to 100 s if the solver finds a feasible solution and arecontinued otherwise.
Revised: Runs are limited to 100 s if the solver finds a feasible solution and are continued otherwise.

Original: Moreover, we set heuristics = 0 . 2 and activate the non-convex hyperparameter for benchmarkproblems with non-convex constraints.
Revised: Moreover, we set heuristics = 0 . 2 and activate the non-convex hyperparameter for benchmark problems with non-convex constraints.

Original: LEAF-GP-RND uses a sampling-based strategy that randomlyevaluates the acquisition function at 2000 locations and selects the minimum value.
Revised: LEAF-GP-RND uses a sampling-based strategy that randomly evaluates the acquisition function at 2000 locations and selects the maximimum value.

Original: C.2 GA
Revised: C.2 GA

Original: We use the standard Genetic Algorithm implementation of the pymoo [4] toolbox for evolutionaryalgorithms and change population_size = 10 given the small evaluation budget.
Revised: We use the standard Genetic Algorithm implementation of the pymoo [7] toolbox for evolutionary algorithms and change population_size = 10 given the small evaluation budget.

Original: Default values areused for all other hyperparameters.
Revised: Default values are used for all other hyperparameters.

Original: C.3 SKOPT-GBRT and SKOPT-RF
Revised: C.3 SKOPT-GBRT and SKOPT-RF

Original: We use the default implementation of Scikit-Optimize [24] with random forest and gradient-boostedtrees base estimators for SKOPT-RF and SKOPT-GBRT , respectively.
Revised: We use the default implementation of Scikit-Optimize [31] with random forest and gradient-boosted trees base estimators for SKOPT-RF and SKOPT-GBRT , respectively.

Original: Default values are used for allhyperparameters.
Revised: Default values are used for all hyperparameters.

Original: C.4 SMAC
Revised: C.4 SMAC

Original: For SMAC [27] we utilize the most recent Python implementation SMAC3 [36] using random forestmodels.
Revised: For SMAC [34] we utilize the most recent Python implementation SMAC3 [44] using random forest models.

Original: Moreover, we specify the hyperparameters run_obj =
Revised: Moreover, we specify the hyperparameters run_obj =

Original: ‘ quality ’ and activate the deterministic flag to allow for reproducibility.
Revised: ‘ quality ’ and activate the deterministic flag to allow for reproducibility.

Original: The Section 5.4 CIFAR-NAS and Section F VAE-NAS benchmarkproblems exhibit hierarchical search space relationships, i.e., hyperparameters of a specific layer areonly relevant if the layer is active.
Revised: The Section 5.4 CIFAR-NAS and Section F VAE-NAS benchmark problems exhibit hierarchical search space relationships, i.e., hyperparameters of a specific layer are only relevant if the layer is active.

Original: We use SMAC ’s InCondition function which allows for certainchild features to be considered only if some parent features have certain values, e.g., the stride oflayer n is only considered if number of layers is at least n .
Revised: We use SMAC ’s InCondition function which allows for certain child features to be considered only if some parent features have certain values, e.g., the stride of layer n is only considered if number of layers is at least n .

Original: This allows SMAC to capture hierarchicalrelationships explicitly and to avoid evaluating multiple equivalent configurations.
Revised: This allows SMAC to capture hierarchical relationships explicitly and to avoid evaluating multiple equivalent configurations.

Original: Default values areused for all other hyperparameters.
Revised: Default values are used for all other hyperparameters.

Original: C.5 UCB-MATERN and EI-MATERN
Revised: C.5 UCB-MATERN and EI-MATERN

Original: The UCB-MATERN and EI-MATERN algorithms use the standard upper confidence bound and expectedimprovement implementations of BoTorch [1].
Revised: The UCB-MATERN and EI-MATERN algorithms use the standard upper confidence bound and expected improvement implementations of BoTorch [2].

Original: Before fitting a GP, we normalize data features andstandardize data outputs.
Revised: Before fitting a GP, we normalize data features and standardize data outputs.

Original: The upper confidence acquisition hyperparameter β is set to 1.96.
Revised: The upper confidence acquisition hyperparameter β is set to 1.96.

Original: We negatethe target values and define raw_samples = 200 for the acquisition function maximization.
Revised: We negate the target values and define raw_samples = 200 for the acquisition function maximization.

Original: Forunconstrained cases, the acquisition optimizer uses 100 restarts.
Revised: For unconstrained cases, the acquisition optimizer uses 100 restarts.

Original: However, for constrained problemswe limit the optimizer to five restarts due to extensive run-times caused by finding feasible solutions.
Revised: However, for constrained problems we limit the optimizer to five restarts due to extensive run-times caused by finding feasible solutions.

Original: Default values are used for all other hyperparameters.
Revised: Default values are used for all other hyperparameters.

Original: D Benchmark problems 
Revised: D Benchmark problems D.1 Unconstrained problems

Original: D.1 Unconstrained problems

Original: Figure 3 shows the results for Hartmann (6D), Rastrigin (10D), Schwefel (10D) and StyblinskiTang (10D) benchmark functions implemented according to Surjanovic and Bingham [51].
Revised: Figure 3 shows the results for Hartmann (6D), Rastrigin (10D), Schwefel (10D) and StyblinskiTang (10D) benchmark functions implemented according to Surjanovic and Bingham [63].

Original: Table 1summarizes number of dimensions and domain evaluated for unconstrained benchmark problems.
Revised: Tablesummarizes number of dimensions and domain evaluated for unconstrained benchmark problems.

Original: [Table]
Revised: [Table]

Revised: ∈ [ − 5 . 0 ,

Revised: [Equation]

Original: D.2 Constrained problems
Revised: D.2 Constrained problems

Original: Fig.
Revised: Fig.

Original: 8 presents results of benchmark problems with known constraints.
Revised: 8 presents results of benchmark problems with known constraints.

Original: Domain bounds withoutdecimals indicate integer-valued variable types.
Revised: Domain bounds without decimals indicate integer-valued variable types.

Original: Benchmark examples G1, G3, G4, G6, G7 and
Revised: Benchmark examples G1, G3, G4, G6, G7 and

Original: G10 are implemented according to Hedar [25].
Revised: G10 are implemented according to Hedar [32].

Original: The Alkylation benchmark [48] is adapted from anopen-source implementation [50].
Revised: The Alkylation benchmark [60] is adapted from an open-source implementation [62].

Original: To compare methods that do not support specific input constraints,we penalize the black-box function output according to:
Revised: To compare methods that do not support specific input constraints, we penalize the black-box function output according to:

Original: [Equation]
Revised: [Equation]

Original: where g ( x ) ≤ 0 and h ( x ) = 0 are inequality and equality constraints, respectively.
Revised: where g ( x ) ≤ 0 and h ( x ) = 0 are inequality and equality constraints, respectively.

Original: Minimizing thecombined black-box output allows methods to find feasible solutions.
Revised: Maximizing the combined black-box output allows methods to find feasible solutions.

Original: Eq.
Revised: Eq.

Original: (15) has the hyperparameterλ which weights the penalty, we test values λ ∈ { 1 , 10 , 100 } and only plot the best run for all methodsthat rely on constraint penalization.
Revised: (15) has the hyperparameter λ which weights the penalty, we test values λ ∈ { 1 , 10 , 100 } and only plot the best run for all methods that rely on constraint penalization.

Original: In the tests conducted, LEAF-GP fully supports explicit inputconstraints.
Revised: In the tests conducted, LEAF-GP fully supports explicit input constraints.

Original: UCB-MATERN and EI-MATERN support linear inequality and equality constraints only.
Revised: UCB-MATERN and EI-MATERN support linear inequality and equality constraints only.

Original: The evolutionary algorithm GA has built-in constraint consideration but does not guarantee feasiblesolutions.
Revised: The evolutionary algorithm GA has built-in constraint consideration but does not guarantee feasible solutions.

Original: Other methods, i.e. LEAF-GP-RND , SKOPT-GBRT , SKOPT-RF and SMAC, rely on the Eq.
Revised: LEAF-GP-RND , SKOPT-GBRT , SKOPT-RF and SMAC rely on the Eq.

Original: (15)penalty function.
Revised: (15) penalty function.

Original: D.3 CIFAR-NAS
Revised: D.3 CIFAR-NAS

Original: Table 3 gives more details on the Section 5.4 CIFAR-NAS (29D) benchmark problem.
Revised: Table 3 gives more details on the Section 5.4 CIFAR-NAS (29D) benchmark problem.

Original: CIFARNAS (29D) has a total of 29 hyperparameters to tune, i.e. 1 continuous, 15 integer, 8 binary and
Revised: CIFARNAS (29D) has a total of 29 hyperparameters to tune, i.e., 1 continuous, 15 integer, 8 binary and

Original: [Table]
Revised: [Table]

Original: ] , x { 2 , 3 } ∈ [10 . 0 , 200 . ]
Revised: ] , x { 2 , 3 } ∈ [10 . 0 , 200 . ]

Original: 5 categorical variables.
Revised: 5 categorical variables.

Original: The goal is to select hyperparameter values for a CNN trained in PyTorch[43] on the CIFAR-10 dataset [33] that maximize test accuracy.
Revised: The goal is to select hyperparameter values for a CNN trained in PyTorch [55] on the CIFAR-10 dataset [40] that maximize test accuracy.

Original: The training and test scripts wereadapted from Trencseni [56].
Revised: The training and test scripts were adapted from Trencseni [70].

Original: Due to limited computing resources, we train the CNN on half of thetraining data for 10 epochs using the Adam solver [31].
Revised: Due to limited computing resources, we train the CNN on half of the training data for 10 epochs using the Adam solver [38].

Original: We score networks using the full test data set.
Revised: We score networks using the full test data set.

Original: Only certain combinations of stride, padding and filter size for various layers result in feasible neuralarchitectures, e.g. the filter size of one layer may be too large given the output of the previous layer.
Revised: Only certain combinations of stride, padding and filter size for various layers result in feasible neural architectures, e.g., the filter size of one layer may be too large given the output of the previous layer.

Original: Insuch cases the CNN training fails, and the black-box returns the largest black-box value found so far,helping algorithms learn to avoid infeasible neural architectures.
Revised: In such cases the CNN training fails, and the black-box returns the largest black-box value found so far, helping algorithms learn to avoid infeasible neural architectures.

Original: To simplify the training, layer inputsare parameterized based on outputs of the previous layer for convolutional and fully-connected layers,as well as the intermediate connecting layer.
Revised: To simplify the training, layer inputs are parameterized based on outputs of the previous layer for convolutional and fully-connected layers, as well as the intermediate connecting layer.

Original: The benchmark introduces categorical variables foractivation function selection.
Revised: The benchmark introduces categorical variables for activation function selection.

Original: Methods that do not support categorical features use one-hot encoding.
Revised: Methods that do not support categorical features use one-hot encoding.

Original: LEAF-GP has access to constraints capturing feasible neural architectures mainly concerned withthe convolutional layers.
Revised: LEAF-GP has access to constraints capturing feasible neural architectures mainly concerned with the convolutional layers.

Original: Algorithms can choose to activate at most three convolutional and twofully-connected layers.
Revised: Algorithms can choose to activate at most three convolutional and two fully-connected layers.

Original: To capture constraints for feasible CNNs, we introduce w out ,i as the output ofconvolutional layer i and W out ,i as the layer’s modified output in case max-pooling is applied:
Revised: To capture constraints for feasible CNNs, we introduce w out ,i as the output of convolutional layer i and W out ,i as the layer’s modified output in case max-pooling is applied:

Original: [Equation]
Revised: [Equation]

Original: 16a)
Revised: 16a)

Original: [Equation]
Revised: [Equation]

Original: 16b)
Revised: 16b

Revised: [Equation]

Original: Convolutional layers use PyTorch’s Conv2D with inputs derived by the optimization algorithms.
Revised: Convolutional layers use PyTorch’s Conv2D with inputs derived by the optimization algorithms.

Original: PyTorch’s MaxPool2d implements the max-pooling with the commonly-used (2 , 2) kernel size.
Revised: PyTorch’s MaxPool2d implements the max-pooling with the commonly-used (2 , 2) kernel size.

Original: [Equation]
Revised: [Equation]

Original: (17a
Revised: 17a)

Original: [Equation]
Revised: [Equation]

Revised: 17b

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The 32 × 32 image size of CIFAR-10 data defines the input to the full CNN ( W in , 1 ) in Eq.
Revised: The 32 × 32 image size of CIFAR-10 data defines the input to the full CNN ( W in , 1 ) in Eq.

Original: (17a).
Revised: (17a).

Original: Eq.
Revised: Eq.

Original: (17b) combines filter size F 1 , padding P 1 and stride S 1 of the first convolutional layer tocompute its output size w out , 1 .
Revised: (17b) combines filter size F 1 , padding P 1 and stride S 1 of the first convolutional layer to compute its output size w out , 1 .

Original: Variable W out , 1 captures the final output of the convolutional layerby considering if the layer is activated, i.e., b conv= 1 , and if max-pooling is applied, i.e., b pool= 1 .
Revised: Variable W out , 1 captures the final output of the convolutional layer by considering if the layer is activated, i.e., b conv= 1 , and if max-pooling is applied, i.e., b pool= 1 .

Original: Constraints (17c)–(17g) denote the same restrictions for subsequent layers, each using the output sizeof the previous layer as its input size.
Revised: Constraints (17c)–(17g) denote the same restrictions for subsequent layers, each using the output size of the previous layer as its input size.

Original: Eq.
Revised: Eq.

Original: (17h) ensures that the output of the last convolutional layer
Revised: (17h) ensures that the output of the last convolutional layer

Original: W out , 3 is at least one.
Revised: W out , 3 is at least one.

Original: We also enforce that at least one layer be active, which is captured by Eq. (17i).
Revised: We also enforce that at least one layer be active, which is captured by Eq. (17i).

Original: To break symmetries in the benchmark problems, we introduce Constraints (18a)–(18g): 
Revised: To break symmetries in the benchmark problems, we introduce Constraints (18a)–(18g): [Equation] conv [Equation]

Revised: conv

Original: [Equation]
Revised: [Equation]

Revised: conv

Revised: [Equation]

Original: (18a)
Revised: (18a)

Original: [Equation]
Revised: [Equation]

Original:  (18b) 
Revised: 3 } (18b) ¬

Original: [Equation]
Revised: [Equation]

Original: (18c
Revised: (18c)

Revised: ¬

Original: [Equation]
Revised: [Equation]

Original: (18d
Revised: 3 } (18d) [Equation] [Equation] 3 } (18e) ¬ b conv

Revised: i

Original: [Equation]
Revised: [Equation]

Original: Eq. (18a) activates layers in a particular order, and Eq.
Revised: Eq. (18a) activates layers in a particular order, and Eq.

Original: (18b) deactivates max-pooling when theassociated convolutional layer is inactive.
Revised: (18b) deactivates max-pooling when the associated convolutional layer is inactive.

Original: Constraints (18c)–(18g) set layer-specific hyperparametersto pre-defined default values when the associated layer is inactive.
Revised: Constraints (18c)–(18g) set layer-specific hyperparameters to pre-defined default values when the associated layer is inactive.

Original: We select these defaults as thelower bound for non-categorical variables and the first category for categorical variables.
Revised: We select these defaults as the lower bound for non-categorical variables and the first category for categorical variables.

Original: Constraints (19) express the same restrictions for fully-connected layers:
Revised: Constraints (19) express the same restrictions for fully-connected layers:

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: pool= x integer [2 , 5]
Revised: pool= x integer [2 , 5]

Original: F 3 = x integer [1 , 3]
Revised: F 3 = x integer [1 , 3]

Original: S 3 = x integer [0 , 3]
Revised: S 3 = x integer [0 , 3]

Original: P 3 = x categ.
Revised: P 3 = x categ.

Original: { ReLU , PReLU , Leaky ReLU }
Revised: { ReLU , PReLU , Leaky ReLU }

Original: Act conv= x binary { 0 , 1 } b fc 1 = x integer [2 , 7]
Revised: Act conv= x binary { 0 , 1 } b fc 1 = x integer [2 , 7]

Original: N fc 1 = 2 x categ.
Revised: N fc 1 = 2 x categ.

Original: { ReLU , PReLU , Leaky ReLU }
Revised: { ReLU , PReLU , Leaky ReLU }

Original: Act fc 1 = x binary { 0 , 1 } b fc 2 = x
Revised: Act fc 1 = x binary { 0 , 1 } b fc 2 = x

Original: Number integer [2 , 7]
Revised: Number integer [2 , 7]

Original: N fc 2 = 2 x categ.
Revised: N fc 2 = 2 x categ.

Original: { ReLU , PReLU , Leaky ReLU }
Revised: { ReLU , PReLU , Leaky ReLU }

Original: Act fc 2 = x
Revised: Act fc 2 = x

Original: E Additional Results
Revised: E Additional Results

Original: This section presents additional results supporting the numerical evaluation in Section
Revised: This section presents additional results supporting the numerical evaluation in Section

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Original: R e l a t i v e 
Revised: l a t i v e P r e d i

Original: P r e d i c t i o n 
Revised: c t i o n [Figure]

Original: E rr o r
Revised: o r

Original: [Figure]
Revised: [Figure]

Original: l a t i ve
Revised: t i v

Original: [Figure]
Revised: [Figure]

Original: r
Revised: r

Original: [Figure]
Revised: [Figure]

Original: i
Revised: i

Original: [Figure]
Revised: [Figure]

Original: t io n
Revised: t i

Original: [Figure]
Revised: [Figure]

Original: rr o r
Revised: n [Figure] [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: a c k
Revised: o x O

Original: [Figure]
Revised: [Figure]

Original: x
Revised: j

Original: [Figure]
Revised: [Figure]

Original: j e c t i v 
Revised: c t i v e

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure][Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: subtitles the number of dimensions ( and inequality constraints ( Section 5.4 provides details.

Original: Table 4 gives more details on the Section 5.4 Variational Autoencoder Neural Architecture Search(VAE-NAS) benchmark problem, which was adapted from Daxberger et al.
Revised: Table 4 gives more details on the Section 5.4 Variational Autoencoder Neural Architecture Search (VAE-NAS) benchmark problem, which was adapted from Daxberger et al.

Original: As their exactimplementation is not publicly available, we created a benchmark problem based on the paperdescription and training scripts by Rath [46].
Revised: As their exact implementation is not publicly available, we created a benchmark problem based on the paper description and training scripts by Rath [58].

Original: VAE-NAS (32D) has a total of 32 hyperparameters totune, i.e., 1 continuous, 20 integer, and 11 categorical variables.
Revised: VAE-NAS (32D) has a total of 32 hyperparameters to tune, i.e., 1 continuous, 20 integer, and 11 categorical variables.

Original: The goal is to select hyperparametervalues for a variational autoencoder (VAE) [32] trained in PyTorch [43] on the MNIST dataset [12]that minimize test loss, i.e., the average loss of when encoding and decoding images from the test set.
Revised: The goal is to select hyperparameter values for a variational autoencoder (VAE) [39] trained in PyTorch [55] on the MNIST dataset [17] that minimize test loss, i.e., the average loss of when encoding and decoding images from the test set.

Original: We train each tested VAE for 32 epochs using the Adam solver [31] and a fixed batch size of 128.
Revised: We train each tested VAE for 32 epochs using the Adam solver [38] and a fixed batch size of 128.

Original: Only certain combinations of stride, padding, and filter size for various layers result in feasible neuralarchitectures.
Revised: Only certain combinations of stride, padding, and filter size for various layers result in feasible neural architectures.

Original: The convolutional layer output size W e out is computed as:
Revised: The convolutional layer output size W e out is computed as:

Original: [Equation]
Revised: [Equation]

Original: Decoder constraints are more complicated given that the size of the VAE output must match theoriginal MNIST image size.
Revised: Decoder constraints are more complicated given that the size of the VAE output must match the original MNIST image size.

Original: The output of deconvolutional layers is computed according to:
Revised: The output of deconvolutional layers is computed according to:

Original: [Equation]
Revised: [Equation]

Original: Matching the output and input sizes of the VAE is non-trivial, as hyperparameters of the convolutionaland deconvolutional layers are themselves set by the optimization algorithms.
Revised: Matching the output and input sizes of the VAE is non-trivial, as hyperparameters of the convolutional and deconvolutional layers are themselves set by the optimization algorithms.

Original: To simplify thetraining, layer inputs are parameterized based on outputs of the previous layer for convolutional,deconvolutional, and fully-connected layers.
Revised: To simplify the training, layer inputs are parameterized based on outputs of the previous layer for convolutional, deconvolutional, and fully-connected layers.

Original: Specifically, fully-connected layers FC2 and FC3 have2 × N lat nodes.
Revised: Specifically, fully-connected layers FC2 and FC3 have 2 × N lat nodes.

Original: According to Daxberger et al.
Revised: According to Daxberger et al.

Original: [11], we parameterize the size of the last fully-connectedlayer—which can be FC3 , FC4 , or the latent space layer depending on which layers are active—as
Revised: [16], we parameterize the size of the last fully-connected layer—which can be FC3 , FC4 , or the latent space layer depending on which layers are active—as

Original: C d 1 × 7 × 7 .
Revised: C d 1 × 7 × 7 .

Original: This allows methods not supporting explicit input constraints to still easily find a feasibleneural architecture by deactivating all deconvolutional layers and computing an output 16 × 7 × 7 =784 with C d 1 = 16 (the original MNIST image size is 1 × 28 × 28 = 784 ).
Revised: This allows methods not supporting explicit input constraints to still easily find a feasible neural architecture by deactivating all deconvolutional layers and computing an output 16 × 7 × 7 = 784 with C d 1 = 16 (the original MNIST image size is 1 × 28 × 28 = 784 ).

Original: This rule supersedes otherlayer size definitions for fully-connected layers.
Revised: This rule supersedes other layer size definitions for fully-connected layers.

Original: Moreover, we extend the benchmark by introducingthe nonlinear activation function of each layer as a categorical optimization hyperparameter.
Revised: Moreover, we extend the benchmark by introducing the nonlinear activation function of each layer as a categorical optimization hyperparameter.

Original: Methodsthat do not support categorical features use one-hot encoding.
Revised: Methods that do not support categorical features use one-hot encoding.

Original: The activation function of the lastlayer is fixed as the sigmoid function, superseding other activation function hyperparameters.
Revised: The activation function of the last layer is fixed as the sigmoid function, superseding other activation function hyperparameters.

Original: The
Revised: The

Original: VAE is trained using the sum of binary cross-entropy loss (reconstruction error) and KL divergence.
Revised: VAE is trained using the sum of binary cross-entropy loss (reconstruction error) and KL divergence.

Original: We use the same loss function to evaluate the VAE’s performance on the test dataset, giving theblack-box objective in Fig. 9.
Revised: We use the same loss function to evaluate the VAE’s performance on the test dataset, giving the black-box objective in Fig. 9.

Original: To allow for a fair comparison, the same 10 randomly sampled feasiblearchitectures initialize all methods.
Revised: To allow for a fair comparison, the same 10 randomly sampled feasible architectures initialize all methods.

Original: LEAF-GP has access to constraints describing feasible neural architectures.
Revised: LEAF-GP has access to constraints describing feasible neural architectures.

Original: We begin by definingauxiliary variables similar to the benchmark in Section D.3:
Revised: We begin by defining auxiliary variables similar to the benchmark in Section D.3:

Original: [Equation]
Revised: [Equation]

Original: w 
Revised: w eout ,i ∈ N 0 , ∀ i ∈ [1 , 2] , (22b) W eout

Original: [Equation]
Revised: [Equation]

Original: ∀ i ∈ 
Revised: i ∈ N

Original: [Equation]
Revised: [Equation]

Original: 22b [Equation] ∀ i ∈ [
Revised: ∀ i ∈ [1 , 2] , (22c) b ecovn

Original: [Equation]
Revised: [Equation]

Original: 22c

Original: [Equation]
Revised: [Equation]

Original: ∀ i ∈ [ [Equation] 22d)
Revised: ∀ i ∈ [1 , 2] , (22d)

Revised: N econv

Original: [Equation]
Revised: [Equation]

Original: 22e)
Revised: (22e)

Original: [Equation]
Revised: [Equation]

Original: 22f) where W ein ,i and W eout ,i denote, respectively, the input and output sizes of convolutional layer i inthe encoder.
Revised: (22f) where W ein ,i and W eout ,i denote, respectively, the input and output sizes of convolutional layer i in the encoder.

Original: We also define an auxiliary variable w eout ,i to track the output size for inactive layers,as well as binary variables b econv ,i corresponding to the active/inactive state of each layer.
Revised: We also define an auxiliary variable w eout ,i to track the output size for inactive layers, as well as binary variables b econv ,i corresponding to the active/inactive state of each layer.

Original: Eq. (22e)and Eq.
Revised: Eq. (22e) and Eq.

Original: (22f) link binary variables
Revised: (22f) link binary variables

Original: b econv ,i to the number of active convolutional layers in the encoder.
Revised: b econv ,i to the number of active convolutional layers in the encoder.

Original: Using these auxiliary variables, the following relations can be expressed:
Revised: Using these auxiliary variables, the following relations can be expressed:

Original: [Equation]
Revised: [Equation]

Original: 23a)

Original: [Equation] [Equation] 23e) Eq. (23a) and Eq. (23b) define the input sizes as the MNIST image input size W ein , 1 = 28 for the firstlayer and the output size of the previous convolution for ensuing layers.
Revised: Eq. (23a) and Eq. (23b) define the input sizes as the MNIST image input size W ein , 1 = 28 for the first layer and the output size of the previous convolution for ensuing layers.

Original: Eq.
Revised: Eq.

Original: (23c) defines the layeri output w eout ,i given the filter size F e i , padding P e i , and stride S e i .
Revised: (23c) defines the layer i output w eout ,i given the filter size F e i , padding P e i , and stride S e i .

Original: Eq.
Revised: Eq.

Original: (23d) ensures that the actualconvolutional layer output W eout ,i only takes the value of w eout ,i if the layer is active.
Revised: (23d) ensures that the actual convolutional layer output W eout ,i only takes the value of w eout ,i if the layer is active.

Original: Finally, Eq.
Revised: Finally, Eq.

Original: (23e)enforces the output size of the encoder to be at least one.
Revised: (23e) enforces the output size of the encoder to be at least one.

Original: We add similar auxiliary variables and constraints for the deconvolutional layers:
Revised: We add similar auxiliary variables and constraints for the deconvolutional layers:

Original: [Equation]
Revised: [Equation]

Original: The Eq.
Revised: The Eq.

Original: (24g) indicator constraint restricts the decoder output to be the original image size
Revised: (24g) indicator constraint restricts the decoder output to be the original image size

Original: W dout , 2 = 28 if deconvolutional layers are active.
Revised: W dout , 2 = 28 if deconvolutional layers are active.

Original: Another indicator constraint Eq.
Revised: Another indicator constraint Eq.

Original: (24h) handlesthe aforementioned case where no deconvolutional layer is active and C d 1 = 16 ensures that thedecoder output size can be resized to original MNIST image size.
Revised: (24h) handles the aforementioned case where no deconvolutional layer is active and C d 1 = 16 ensures that the decoder output size can be resized to original MNIST image size.

Original: We emphasize that this rule isintroduced to simplify the feasible architecture search for methods that do not support explicit inputconstraints.
Revised: We emphasize that this rule is introduced to simplify the feasible architecture search for methods that do not support explicit input constraints.

Original: Note that LEAF-GP could add additional constraints to ensure the architecture’s outputsize can always be resized to the original image size of 28 × 28. 
Revised: Note that LEAF-GP could add additional constraints to ensure the architecture’s output size can always be resized to the original image size of 28 × 28. [Equation] din ,i ,

Original: [Equation]
Revised: [Equation]

Original: Similar to the encoder, Eq.
Revised: Similar to the encoder, Eq.

Original: (25) defines constraints for feasible decoder layers.
Revised: (25) defines constraints for feasible decoder layers.

Original: For deconvolutionallayers, we also tune output padding O d i .
Revised: For deconvolutional layers, we also tune output padding O d i .

Original: According to the PyTorch [18] documentation, outputpadding must be smaller than either stride or dilation.
Revised: According to the PyTorch [24] documentation, output padding must be smaller than either stride or dilation.

Original: Given that we do not optimize dilation in deconvolutional layers, Eq. (25d) enforces output padding to be smaller than stride.
Revised: Given that we do not optimize dilation in deconvolutional layers, Eq. (25d) enforces output padding to be smaller than stride.

Original: We introducesimilar constraints for fully-connected layers in both the encoder and decoder:
Revised: We introduce similar constraints for fully-connected layers in both the encoder and decoder:

Original: [Equation]
Revised: [Equation]

Original: 26a [Equation] 26e) To break symmetries in the benchmark problem, we add constraints (27a)–(27m):
Revised: To break symmetries in the benchmark problem, we add constraints (27a)–(27m):

Original: [Equation]
Revised: [Equation]

Original: , ∀ i ∈ [1 , 2] , (27a)¬ b econv
Revised: ∀ i ∈ [1 , 2] , (27a)

Original: [Equation]
Revised: [Equation]

Original: i → S e i ≤ 1 , ∀ i ∈ [1 , 2] , (27b)
Revised: ∀ i ∈ [1 , 2] , (27b)

Original: ¬ b econv ,i → P e i ≤ 0 , ∀ i ∈ [1 , 2] , (27c) ¬ b econv ,i → F e i = 2 , ∀ i ∈ [1 , 2] , (27d)
Revised: [Equation] ∀ i ∈ [1 , 2] , (27c) [Equation] ∀ i ∈ [1 , 2] , (27d)

Original: ¬ b econv ,i → Act e i = ReLU , ∀ i ∈
Revised: [Equation]

Original:  [1 , 2] , (27e)¬
Revised: , ∀ i ∈ [1 , 2] , (27e)

Original: [Equation]
Revised: [Equation]

Original: fc ,i → Act fc i = ReLU , ∀ i ∈ [1 , 4] , (27f) 
Revised: ∀ i ∈ [1 , 4] , (27f) [Equation] [Equation]

Original: ¬ b fc , 1 → N fc 1 ≤ 0 , (27g)
Revised: (27g) [Equation] ∀ i ∈ [1 , 2] , (27h) [Equation]

Original: ¬ b ddec ,i → C d i ≤ 4 , ∀ i ∈ [1 , 2] , (27h)
Revised: ddec [Equation] i [Equation]

Original: ¬ b ddec ,i → S d i ≤ 1 , ∀ i ∈ [1 , 2] , (27i) 
Revised: i [Equation] [Equation] ∀ i ∈ [1 , 2] , (27i) ¬

Revised: [Equation]

Original: ¬ b ddec ,i → P d i ≤ 0 , ∀ i ∈ [1 , 2] , (27j)
Revised: ddec , i [Equation] P d i ≤ 0 , ∀ i ∈ [1 , 2] , (27j)

Revised: [Equation]

Original: ¬ b ddec ,i → O d i ≤ 0 , ∀ i ∈ [1 , 2] , (27k)
Revised: ddec , i [Equation] O d i ≤ 0 , ∀ i ∈ [1 , 2] , (27k)

Revised: [Equation]

Original: ¬ b ddec ,i → F d i = 2 , ∀ i ∈ [1 , 2] , (27l)
Revised: ddec , i [Equation] F d i = 2 , ∀ i ∈ [1 , 2] , (27l)

Revised: ¬

Original: ¬ b ddec , 1 → Act d 1 = ReLU(27m)
Revised: [Equation] ddec , [Equation] d 1 = ReLU

Original:  Constraints (27a)–(27m) set layer-specific hyperparameters to pre-defined default values when theassociated layer is inactive.
Revised: [Equation] Constraints (27a)–(27m) set layer-specific hyperparameters to pre-defined default values when the associated layer is inactive.

Original: We select these defaults as the lower bound for non-categorical variablesand the first category for categorical variables.
Revised: We select these defaults as the lower bound for non-categorical variables and the first category for categorical variables.

Original: While SMAC is unable to handle more complicatedconstraints restricting outputs of deconvolutional layers, it can handle hierarchical search spacestructures.
Revised: While SMAC is unable to handle more complicated constraints restricting outputs of deconvolutional layers, it can handle hierarchical search space structures.

Original: For VAE-NAS benchmark runs using SMAC as an optimizer we enforce hierarchiesaccording to constraints (27a)–(27m) which deactivate hyperparameters for inactive layers.
Revised: For VAE-NAS benchmark runs using SMAC as an optimizer we enforce hierarchies according to constraints (27a)–(27m) which deactivate hyperparameters for inactive layers.

Original: The transformation column refers to post-processing computations before passing the hyperparameter value to the neural network training.
Revised: The transformation column refers to post-processing computations before passing the hyperparameter value to the neural network training.

Original: The architecture with all layers activated comprises
Revised: The architecture with all layers activated comprises

Original: C1-C2-FC1-FC2-L-FC3-FC4-D1-D2 , with L referring to the latent space layer.
Revised: C1-C2-FC1-FC2-L-FC3-FC4-D1-D2 , with L referring to the latent space layer.

Original: [Table]
Revised: [Table]

Original: conti.
Revised: conti.

Original: [ − 4 . 0 , − 2 . 0]
Revised: [ − 4 . 0 , − 2 . 0]

Original: α = 10 x
Revised: α = 10 x

Original: [Table]
Revised: [Table]

Original: integer [16 , 64]
Revised: integer [16 , 64]

Original: N lat
Revised: N lat

Original: = x
Revised: = x

Original: [Table]
Revised: [Table]

Original: integer [0 , 2]
Revised: integer [0 , 2]

Original: N econv
Revised: N econv

Original: =
Revised: =

Original: x
Revised: x

Original: [Table]
Revised: [Table]

Original: integer [0 , 2]
Revised: integer [0 , 2]

Original: N efc
Revised: N efc

Original: = x
Revised: = x

Original: [Table]
Revised: [Table]

Original: integer [0 , 2]
Revised: integer [0 , 2]

Original: N ddec =
Revised: N ddec =

Original: x
Revised: x

Original: [Table]
Revised: [Table]

Original: layers integer [0 , 2]
Revised: layers integer [0 , 2]

Original: N dfc
Revised: N dfc

Original: =
Revised: =

Original: x
Revised: x

Original: [Table]
Revised: [Table]

Original: integer [2 , 5]
Revised: integer [2 , 5]

Original: C e 1 = 2 x
Revised: C e 1 = 2 x

Original: [Table]
Revised: [Table]

Original: integer [1 , 2]
Revised: integer [1 , 2]

Original: S e 1 = x
Revised: S e 1 = x

Original: [Table]
Revised: [Table]

Original: integer [0 , 3]
Revised: integer [0 , 3]

Original: P e 1 = x
Revised: P e 1 = x

Original: [Table]
Revised: [Table]

Original: categ.
Revised: categ.

Original: F e 1 = x
Revised: F e 1 = x

Original: [Table]
Revised: [Table]

Original: categ.
Revised: categ.

Original: { ReLU , PReLU , Leaky ReLU }
Revised: { ReLU , PReLU , Leaky ReLU }

Original: Act e 1 = x
Revised: Act e 1 = x

Original: [Table]
Revised: [Table]

Original: integer [3 , 6]
Revised: integer [3 , 6]

Original: C e 2 = 2 x
Revised: C e 2 = 2 x

Original: [Table]
Revised: [Table]

Original: integer [1 , 2]
Revised: integer [1 , 2]

Original: S e 2 = x
Revised: S e 2 = x

Original: [Table]
Revised: [Table]

Original: integer [0 , 3]
Revised: integer [0 , 3]

Original: P e 2 = x
Revised: P e 2 = x

Original: [Table]
Revised: [Table]

Original: categ.
Revised: categ.

Original: F e 2 = x
Revised: F e 2 = x

Original: [Table]
Revised: [Table]

Original: categ.
Revised: categ.

Original: { ReLU , PReLU , Leaky ReLU }
Revised: { ReLU , PReLU , Leaky ReLU }

Original: Act e 2 = x
Revised: Act e 2 = x

Original: Fully-connected layer 1 ( FC1 )
Revised: Fully-connected layer 1 ( FC1 )

Original: Number of nodes integer [0 , 15]
Revised: Number of nodes integer [0 , 15]

Original: N fc 1 = 64 × x
Revised: N fc 1 = 64 × x

Original: Activation function categ.
Revised: Activation function categ.

Original: { ReLU , PReLU , Leaky ReLU }
Revised: { ReLU , PReLU , Leaky ReLU }

Original: Act fc 1 = x
Revised: Act fc 1 = x

Original: Fully-connected layer 2 ( FC2 )
Revised: Fully-connected layer 2 ( FC2 )

Original: Activation function categ.
Revised: Activation function categ.

Original: { ReLU , PReLU , Leaky ReLU }
Revised: { ReLU , PReLU , Leaky ReLU }

Original: Act fc 2 = x
Revised: Act fc 2 = x

Original: Decoder
Revised: Decoder

Original: Fully-connected layer 3 ( FC3 )
Revised: Fully-connected layer 3 ( FC3 )

Original: Activation function categ.
Revised: Activation function categ.

Original: { ReLU , PReLU , Leaky ReLU }
Revised: { ReLU , PReLU , Leaky ReLU }

Original: Act fc 3 = x
Revised: Act fc 3 = x

Original: Fully-connected layer 4 ( FC4 )
Revised: Fully-connected layer 4 ( FC4 )

Original: Activation function categ.
Revised: Activation function categ.

Original: { ReLU , PReLU , Leaky ReLU }
Revised: { ReLU , PReLU , Leaky ReLU }

Original: Act fc 4 = x
Revised: Act fc 4 = x

Original: Deconvolutional layer 1 ( D1 )
Revised: Deconvolutional layer 1 ( D1 )

Original: Number of input channels integer [3 , 6]
Revised: Number of input channels integer [3 , 6]

Original: C d 1 = 2 x
Revised: C d 1 = 2 x

Original: Stride integer [1 , 2]
Revised: Stride integer [1 , 2]

Original: S d 1 = x
Revised: S d 1 = x

Original: Padding integer [0 , 3]
Revised: Padding integer [0 , 3]

Original: P d 1 = x
Revised: P d 1 = x

Original: Output Padding integer [0 , 1]
Revised: Output Padding integer [0 , 1]

Original: O d 1 = x
Revised: O d 1 = x

Original: Filter size categ.
Revised: Filter size categ.

Original: F d 1 = x
Revised: F d 1 = x

Original: Activation function categ.
Revised: Activation function categ.

Original: { ReLU , PReLU , Leaky ReLU }
Revised: { ReLU , PReLU , Leaky ReLU }

Original: Act d 1 = x
Revised: Act d 1 = x

Original: Deconvolutional layer 2 ( D2 )
Revised: Deconvolutional layer 2 ( D2 )

Original: Number of input channels integer [2 , 5]
Revised: Number of input channels integer [2 , 5]

Original: C d 2 = 2 x
Revised: C d 2 = 2 x

Original: Stride integer [1 , 2]
Revised: Stride integer [1 , 2]

Original: S d 2 = x
Revised: S d 2 = x

Original: Padding integer [0 , 3]
Revised: Padding integer [0 , 3]

Original: P d 2 = x
Revised: P d 2 = x

Original: Output Padding integer [0 , 1]
Revised: Output Padding integer [0 , 1]

Original: O d 2 = x
Revised: O d 2 = x

Original: Filter size categ.
Revised: Filter size categ.

Original: F d 2 = x
Revised: F d 2 = x