Original: Tree ensemble kernels for Bayesian optimization with known constraints over mixed-feature spaces
Revised: Tree ensemble kernels for Bayesian optimization with known constraints over mixed-feature spaces

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Tree ensembles can be well-suited for black-box optimization tasks such as algorithm tuning and neural architecture search, as they achieve good predictive performance with little or no manual tuning, naturally handle discrete feature spaces, andare relatively insensitive to outliers in the training data.
Revised: Tree ensembles can be well-suited for black-box optimization tasks such as algorithm tuning and neural architecture search, as they achieve good predictive performance with little or no manual tuning, naturally handle discrete feature spaces, andare relatively insensitive to outliers in the training data.

Original: Two well-known challengesin using tree ensembles for black-box optimization are (i) effectively quantifyingmodel uncertainty for exploration and (ii) optimizing over the piece-wise constantacquisition function.
Revised: Two well-known challengesin using tree ensembles for black-box optimization are (i) effectively quantifyingmodel uncertainty for exploration and (ii) optimizing over the piece-wise constantacquisition function.

Original: To address both points simultaneously, we propose using thekernel interpretation of tree ensembles as a Gaussian Process prior to obtain modelvariance estimates, and we develop a compatible optimization formulation for theacquisition function.
Revised: To address both points simultaneously, we propose using thekernel interpretation of tree ensembles as a Gaussian Process prior to obtain modelvariance estimates, and we develop a compatible optimization formulation for theacquisition function.

Original: The latter further allows us to seamlessly integrate knownconstraints to improve sampling efficiency by considering domain-knowledgein engineering settings and modeling search space symmetries, e.g. hierarchicalrelationships in neural architecture search.
Revised: The latter further allows us to seamlessly integrate knownconstraints to improve sampling efficiency by considering domain-knowledgein engineering settings and modeling search space symmetries, e.g. hierarchicalrelationships in neural architecture search.

Original: Our framework performs as well asstate-of-the-art methods for unconstrained black-box optimization over continuous/discrete features and outperforms competing methods for problems combiningmixed-variable feature spaces and known input constraints.
Revised: Our framework performs as well asstate-of-the-art methods for unconstrained black-box optimization over continuous/discrete features and outperforms competing methods for problems combiningmixed-variable feature spaces and known input constraints.

Original:  Introduction
Revised:  Introduction

Original: Many black-box optimization problems contain feature relationships known a priori based on domainknowledge, such as hierarchies or constraints [40].
Revised: Many black-box optimization problems contain feature relationships known a priori based on domainknowledge, such as hierarchies or constraints [34].

Original: For example, hierarchical structures arise inneural architecture search [15, 58], where hyperparameters such as kernel size are only relevant if aconvolutional layer is selected.
Revised: For example, hierarchical structures arise inneural architecture search [14, 47], where hyperparameters such as kernel size are only relevant if aconvolutional layer is selected.

Original: Explicit constraints can also arise, e g. matching kernel size and strideto input channel size and padding.
Revised: Explicit constraints can also arise, e g. matching kernel size and strideto input channel size and padding.

Original: In many cases, Bayesian optimization can incorporate knownhierarchies and/or constraints, given that a suitable surrogate model is selected.
Revised: In many cases, Bayesian optimization can incorporate knownhierarchies and/or constraints, given that a suitable surrogate model is selected.

Original: To this end, Fromontet al.
Revised: To this end, Fromontet al.

Original: [17] and Nijssen and Fromont [41] impose a variable hierarchy by constraining the splittingorder of decision trees, i.e. certain attributes must be selected before others.
Revised: [16] and Nijssen and Fromont [35] impose a variable hierarchy by constraining the splittingorder of decision trees, i.e. certain attributes must be selected before others.

Original: Tree-based models, such as random forests or gradient-boosted trees, remain popular in many applications, as they inherit the innate ability of simple decision trees to seamlessly handle categorical anddiscrete input spaces.
Revised: Tree-based models, such as random forests or gradient-boosted trees, remain popular in many applications, as they inherit the innate ability of simple decision trees to seamlessly handle categorical and discrete input spaces.

Original: Moreover, they are highly parallelizable and scalable to high-dimensional data.
Revised: Moreover, they are highly parallelizable and scalable to high-dimensional data.

Original: Despite these modelling advantages, the deployment of tree-based models in Bayesian optimization has been limited by challenges pertaining to (i) quantifying prediction uncertainty and (ii) optimizingacquisition functions defined by their discontinuous response surfaces [49].
Revised: Despite these modelling advantages, the deployment of tree-based models in Bayesian optimization has been limited by challenges pertaining to (i) quantifying prediction uncertainty and (ii) optimizingacquisition functions defined by their discontinuous response surfaces [41].

Original: Early works, e.g. thepopular SMAC algorithm [27], addressed (i) using empirical variance within a tree ensemble and (ii)via local and/or random search methods.
Revised: Early works, e.g. thepopular SMAC algorithm [25], addressed (i) using empirical variance within a tree ensemble and (ii)via local and/or random search methods.

Original: Moreover, recent works [38, 39, 53] propose mixed-integerformulations for tree ensembles, enabling optimization over their mean functions.
Revised: Moreover, recent works [32, 33, 43] propose mixed-integerformulations for tree ensembles, enabling optimization over their mean functions

Original: Contributions.
Revised: Contributions.

Original: We present a mixed-integer second-order cone optimization formulation for tree kernel Gaussian processes in Section 4 and show that that the tree agreement ratio , i.e. the hyper- parameter introduced by the tree ensemble kernel, sufficiently represents the model uncertaintyin Section 5.1.
Revised: We present a mixed-integer second-order cone optimization formulation for tree kernel Gaussian processes in Section 3 and show that that the tree agreement ratio , i.e. the hyper- parameter introduced by the tree ensemble kernel, sufficiently represents the model uncertaintyin Section 5.1.

Original: Section 5.2 shows that solving the mixed-integer second-order cone optimizationproblem considerably outperforms sampling-based strategies in Bayesian optimization.
Revised: Section 5.2 shows that solving the mixed-integer second-order cone optimizationproblem considerably outperforms sampling-based strategies in Bayesian optimization.

Original: Our approachof using tree ensemble kernels as a Gaussian process prior is particularly useful for applicationscombining mixed-variable spaces and known input constraints.
Revised: Our approachof using tree ensemble kernels as a Gaussian process prior is particularly useful for applicationscombining mixed-variable spaces and known input constraints.

Original:  Related work
Revised:  Related work

Original: Bayesian optimization (BO) solves [16, 20, 49]: x ∗ f ∈ arg min x f ( x ) , where f is an expensive-toevaluate black-box function that can be queried at inputs x ∈ X to derive the optimal solution x ∗ f .
Revised: Bayesian optimization (BO) solves [15, 18, 41]: x ∗ f ∈ arg min x f ( x ) , where f is an expensive-toevaluate black-box function that can be queried at inputs x ∈ R to derive the optimal solution x ∗ f .

Original: BO iteratively updates a surrogate model of f and optimizes a corresponding acquisition function thatbalances exploitation and exploration.
Revised: BOiteratively updates a surrogate model of f and optimizes a corresponding acquisition function thatbalances exploitation and exploration.

Original: Minimizing the acquisition function produces a new query x ∗which is evaluated and added to the set of observations.
Revised: Minimizing the acquisition function produces a new query x ∗which is evaluated and added to the set of observations.

Original: Gaussian processes (GPs) [45] are a commonchoice of BO surrogate due to their flexibility, e.g. domain-specific knowledge can be built into the
Revised: Gaussian processes (GPs) [38] are a commonchoice of BO surrogate due to their flexibility, e.g. domain-specific knowledge can be built into the

Original: GP prior via mean and kernel functions, and reliable uncertainty quantification to identify unexploredsearch areas.
Revised: GP prior via mean and kernel functions, and reliable uncertainty quantification to identify unexploredsearch areas.

Original: Open-source tools like BoTorch [1] implement BO with GP surrogates and offer a wideselection of kernels mainly suited for continuous search spaces.
Revised: Open-source tools like BoTorch [1] implement BO with GP surrogates and offer a wideselection of kernels mainly suited for continuous search spaces.

Original: Prior works developed GPs withmodified kernels to integrate discrete features [6, 14, 19, 23, 47] or considered conditional featurespaces [22, 28, 35, 37].
Revised: Prior work developed GPs withmodified kernels to integrate discrete features [6, 13, 17, 21, 39] or considered conditional featurespaces [20, 26, 30, 31].

Original: Besides GPs, tree ensemble-based surrogates show excellent performance forblack-box optimization with mixed-variable settings, i.e. with continuous, integer and categoricalvariables, and for structured search spaces, e.g. hierarchical and conditional feature spaces [27].
Revised: Besides GPs, tree ensemble-based surrogates show excellent performance forblack-box optimization with mixed-variable settings, i.e. with continuous, integer and categoricalvariables, and for structured search spaces, e.g. hierarchical and conditional feature spaces [25].

Original: Black-box optimization tools using tree ensembles,
Revised: Black-box optimization tools using tree ensembles,

Original: e.g. SMAC [27] and Scikit-Optimize ( SKOPT ) [24],are useful for applications such as neural architecture search (NAS) and algorithm tuning.
Revised: e.g. SMAC [25] and Scikit-Optimize ( SKOPT ) [22],are useful for applications such as neural architecture search (NAS) and algorithm tuning.

Original: Shahriariet al.
Revised: Shahriariet al.

Original: [49] mention challenges in deploying tree ensembles for BO: (i) quantifying uncertainty forexploration purposes, and (ii) optimizing over the non-differentiable discrete acquisition functionto determine the next query point.
Revised: [41] mention challenges in deploying tree ensembles for BO: (i) quantifying uncertainty forexploration purposes, and (ii) optimizing over the non-differentiable discrete acquisition functionto determine the next query point.

Original: SMAC identifies uncertain search space regions using empiricalvariables across tree predictions of the random forest and optimizes the acquisition function combininglocal and random search.
Revised: SMAC identifies uncertain search space regions using empiricalvariables across tree predictions of the random forest and optimizes the acquisition function combininglocal and random search.

Original: Bergstra et al.
Revised: Bergstra et al.

Original: [3] proposes the Tree Parzen Estimator (TPE) to handlecategorical variables and conditional structures by modeling individual input dimensions by a kerneldensity estimator.
Revised: [3] proposes the Tree Parzen Estimator (TPE) to handlecategorical variables and conditional structures by modeling individual input dimensions by a kerneldensity estimator.

Original: However, the TPE approach ignores dependencies between dimensions.
Revised: However, the TPE approach ignores dependencies between dimensions.

Original: Forgradient-boosted tree ensembles
Revised: Forgradient-boosted tree ensembles

Original: , SKOPT derives uncertainty with quantile regression to fit two modelsfor the 16th and 84th percentile and averages the predictions to estimate the standard deviation.
Revised: , SKOPT derives uncertainty with quantile regression to fit two modelsfor the 16th and 84th percentile and averages the predictions to estimate the standard deviation.

Original: Forrandom forests, SKOPT uses an uncertainty strategy similar to SMAC .
Revised: Forrandom forests, SKOPT uses an uncertainty strategy similar to SMAC .

Original: In general, SKOPT relies onrandom sampling to optimize the acquisition function.
Revised: In general, SKOPT relies onrandom sampling to optimize the acquisition function.

Original: Miši´c [38] proposed a mixed-integer optimization formulation for tree ensemble mean functionsthat has been used in several applications [7, 39, 53, 54].
Revised: Miši´c [32] proposed a mixed-integer optimization formulation for tree ensemble mean functionsthat has been used in several applications [7, 33, 43, 44].

Original: Besides improving the solution to anacquisition function, mixed-integer formulations also allow explicit consideration of input constraintsto incorporate domain-knowledge.
Revised: Besides improving the solution to anacquisition function, mixed-integer formulations also allow explicit consideration of input constraintsto incorporate domain-knowledge.

Original: Some software tools, e.g. BoTorch, support linear equality andinequality constraints of continuous variables at the acquisition function optimization step while treeensemble-based algorithms do not support input constraints.
Revised: Some software tools, e.g. BoTorch, support linear equality andinequality constraints of continuous variables at the acquisition function optimization step while treeensemble-based algorithms do not support input constraints.

Original: Papalexopoulos et al.
Revised: Papalexopoulos et al.

Original: [42] use ReLUneural networks as surrogate models and deploy a mixed-integer linear formulation to optimize theacquisition function.
Revised: [36] use ReLUneural networks as surrogate models and deploy a mixed-integer linear formulation to optimize theacquisition function.

Original: The approach relies on random initialization and stochasticity in the modeltraining to allow for exploration.
Revised: The approach relies on random initialization and stochasticity in the modeltraining to allow for exploration.

Original: Daxberger et al.
Revised: Daxberger et al.

Original: [11] handles mixed-variable search spaces by usinga Bayesian linear regressor that uses an integer solver to search the discrete subspace.
Revised: [11] handles mixed-variable search spaces by usinga Bayesian linear regressor that uses an integer solver to search the discrete subspace.

Original: The authorsintroduce features capturing the discrete parts of the search space by using a BOCS model [2, 13]while continuous parts are handled with random Fourier features [44].
Revised: The authorsintroduce features capturing the discrete parts of the search space by using a BOCS model [2, 12]while continuous parts are handled with random Fourier features [37].

Original: Genetic Algorithms ( GA ) areanother class of algorithms, which deploy evolution-based selection heuristics to minimize black-boxfunctions [29].
Revised: Genetic Algorithms ( GA ) areanother class of algorithms, which deploy evolution-based selection heuristics to minimize black-boxfunctions [27].

Original: While there is no feasibility guarantee for input constraints, GA implementations likepymoo [4] support constraint optimization by minimizing constraint violation.
Revised: While there is no feasibility guarantee for input constraints, GA implementations likepymoo [4] support constraint optimization by minimizing constraint violation.

Original: We compare our BO approach, which uses the kernel interpretation of tree ensembles as a Gaussianprocess prior, to: SMAC , the random forest and gradient-boosted tree versions of SKOPT , i.e. SKOPT-RF ,and SKOPT-GBRT , in the Section 5 numerical studies as a baseline for other tree ensemble-basedalgorithms.
Revised: We compare our BO approach, which uses the kernel interpretation of tree ensembles as a Gaussianprocess prior, to: SMAC , the random forest and gradient-boosted tree versions of SKOPT , i.e. SKOPT-RF ,and SKOPT-GBRT , in the Section 5 numerical studies as a baseline for other tree ensemble-basedalgorithms.

Original: We also compare against the default upper-confidence bound and expected improvement
Revised: We also compare against the default upper-confidence bound BO implementation of

Original: BO implementations of BoTorch, i.e. UCB-MATERNand EI-MATERN , and the default GA algorithm ofpymoo to include black-box algorithms that partially support constraint optimization.
Revised: BoTorch, i.e. UCB-MATERN , and the default GA algorithm of pymoo to include black-box algorithmsthat partially support constraint optimization.

Original: [Figure]
Revised: [Figure]

Original:  Technical background on prior work3.1 Tree ensemble kernel as a Gaussian process prior
Revised:  Technical background on prior work3.1 Tree ensemble kernel as a Gaussian process prior

Original: Exploring the search space requires quantifying the uncertainty of the underlying surrogate model.
Revised: Exploring the search space requires quantifying the uncertainty of the underlying surrogate model.

Original: We use the kernel interpretation of tree ensembles based on random partitions [10, 59].
Revised: We use the kernel interpretation of tree ensembles based on random partitions [10, 48].

Original: The treekernel captures correlation between two input data points ( x , x ′ ) ∈ R n :
Revised: The treekernel captures correlation between two input data points ( x , x ′ ) ∈ R n :

Original: [Equation]
Revised: [Equation]

Original:  To derive the tree kernel, we first train a gradient boosted tree ensemble T on data set X ∈ R m × nwith n denoting the dimensionality of the search space and m the size of the data set.
Revised: [Equation] To derive the tree kernel, we first train a gradient boosted tree ensemble T on data set X ∈ R m × nwith n denoting the dimensionality of the search space and m the size of the data set.

Original: Every tree t inthe tree ensemble T maps inputs x ∈ R n onto a leaf l by sequentially evaluating splitting conditions.
Revised: Every tree t inthe tree ensemble T maps inputs x ∈ R n onto a leaf l by sequentially evaluating splitting conditions.

Original: Each leaf l defines a subspace x l ⊂ R n restricted by active splits s ∈ splits ( t ) .
Revised: Each leaf l defines a subspace x l ∈ R n restricted by active splits s ∈ splits ( t ) .

Original: Two inputs ( x , x ′ )are fully correlated in tree t if both end up in the leaf subspace x t,l and uncorrelated otherwise.
Revised: Two inputs ( x , x ′ )are fully correlated in tree t if both end up in the leaf subspace x t,l and uncorrelated otherwise.

Original: The
Revised: The

Original: Eq.
Revised: Eq.

Original: (1) vector z ( x ) consists of binary elements z t,l indicating if leaf l ∈ L t is active for input x , with
Revised: (1) vector z ( x ) consists of binary elements z t,l indicating if leaf l ∈ L t is active for input x , with

Original: L t denoting the set of all leaves in tree t .
Revised: L t denoting the set of all leaves in tree t .

Original: The inner product z ( x ) ⊺ z ( x ′ ) , normalized by the totalnumber of trees |T | , gives the ratio of trees in the ensemble for which x and x ′ fall into the same leaf.
Revised: The inner product z ( x ) ⊺ z ( x ′ ) , normalized by the totalnumber of trees |T | , gives the ratio of trees in the ensemble for which x and x ′ fall into the same leaf.

Original: We modify the kernel by adding a trainable signal variance σ 20 [34].
Revised: We modify the kernel by adding a trainable signal variance σ 20 [29].

Original: Davies and Ghahramani [10]prove that the tree kernel is a suitable GP prior.
Revised: Davies and Ghahramani [10]prove that the tree kernel is a suitable GP prior.

Original: The resulting (non-stationary and supervised) treekernel describes a prior over piece-wise constant functions when used in a GP.
Revised: The resulting (non-stationary and supervised) treekernel describes a prior over piece-wise constant functions when used in a GP.

Original: 3.2 Posterior distribution
Revised: 3.2 Posterior distribution

Original: We approximate f as a Gaussian process with zero mean and kernel k Tree : f ( · ) ∼ GP (0 , k Tree ) .
Revised: We approximate f as a Gaussian process with zero mean and kernel k Tree : f ( · ) ∼ GP (0 , k Tree ) .

Original: Sincek Tree is a valid Mercer kernel, the mean M ( x ) and variance V ( x ) of the GP at x ∈ R n is [45]:
Revised: Sincek Tree is a valid Mercer kernel, the mean M ( x ) and variance V ( x ) of the GP at x ∈ R n is [38]:

Original: [Equation]
Revised: [Equation]

Original: The Gram matrix K X , X ∈ R m × m has entries describing pairwise correlations computed based on thekernel function in Eq.
Revised: The Gram matrix K X , X ∈ R m × m has entries describing pairwise correlations computed based on thekernel function in Eq.

Original: The entries of vector K x , X ∈ R m contain correlations between the inputx and sampled data points, defined as [ k Tree ( x , x 1 ) , k Tree ( x , x 2 ) , . . .
Revised: The entries of vector K x , X ∈ R m contain correlations between the inputx and sampled data points, defined as [ k Tree ( x , x 1 ) , k Tree ( x , x 2 ) , . . .

Original: , k Tree ( x , x m )] with vectors x ireferring to rows in data set X .
Revised: , k Tree ( x , x m )] with vectors x ireferring to rows in data set X .

Original: Target values
Revised: Target values

Original: y ∈ R m are the corresponding observations for X .
Revised: y ∈ R m are the corresponding observations for X .

Original: Eq.
Revised: Eq.

Original: (2) describes the noise-free case of the GP mean and variance.
Revised: (2) describes the noise-free case of the GP mean and variance.

Original: To fit a GP based on the treekernel function, we usually require a noise term, i.e. a diagonal matrix σ 2 y I that is added to K X , X .
Revised: To fit a GP based on the treekernel function, we usually require a noise term, i.e. a diagonal matrix σ 2 y I that is added to K X , X .

Original: We set hyperparameters σ 2 y and σ 2 0 by maximizing the log marginal likelihood.
Revised: We set hyperparameters σ 2 y and σ 2 0 by maximizing the log marginal likelihood.

Original: Fig.
Revised: Fig.

Original: 1 visualizes the
Revised: 1 visualizes the

Original: Eq.
Revised: Eq.

Original: (2) M ( x ) , V ( x ) and the lower-confidence bound (LCB)
Revised: (2) M ( x ) , V ( x ) and the lower-confidence bound (LCB)

Original: [9] response surface.
Revised: [9] response surface.

Original: Fig.
Revised: Fig.

Original: 1(a) showsthat the tree kernel-based GP mean gives a good piecewise-constant approximation of the Braninfunction.
Revised: 1(a) showsthat the tree kernel-based GP mean gives a good piecewise-constant approximation of the Braninfunction.

Original: Fig. 1(b) shows that variance peaks reveal areas where data are sparse, reliably identifyinguncertainty in the underlying surrogate model.
Revised: Fig. 1(b) shows that variance peaks reveal areas where data is sparse, reliably identifyinguncertainty in the underlying surrogate model.

Original: Finally, Fig.
Revised: Finally, Fig.

Original: 1(c) shows how an acquisition functionsuch as the LCB can effectively manage the exploitation-exploration trade-off.
Revised: 1(c) shows how an acquisition functionsuch as the LCB can effectively manage the exploitation-exploration trade-off.

Original: 3.3 Global optimization of tree ensembles
Revised: 3.3 Global optimization of tree ensembles

Original: Miši´c [38] proposes a mixed-integer linear optimization formulation that ensures that binary variablesz t,l follow the logic of the tree ensemble:
Revised: Miši´c [32] proposes a mixed-integer linear optimization formulation that ensures that binary variablesz t,l follow the logic of the tree ensemble:

Original: [Equation]
Revised: [Equation]

Original: ∀ t ∈ T , ∀ s ∈ splits[Equation] t
Revised: ∀ t ∈ T , ∀ s ∈ splits

Original: [Equation]
Revised: [Equation]

Original: l ∈ right
Revised: l ∈ right

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: ∀ t ∈ T , ∀ s ∈ splits ( t ) , (3c)
Revised: ∀ t ∈ T , ∀ s ∈ splits ( t ) , (3c)

Original: [Equation]
Revised: [Equation]

Original: ∀ i ∈ C , (3d)
Revised: ∀ i ∈ C , (3d)

Original: [Equation]
Revised: [Equation]

Original: ∀ i ∈ N , ∀ j ∈
Revised: ∀ i ∈ N , ∀ j ∈

Original: [ K i − 1]
Revised: [ K i − 1]

Original: [Equation]
Revised: [Equation]

Original: 3e)
Revised: 3e)

Original: [Equation]
Revised: [Equation]

Original: ∀ i ∈ [ n ] , ∀ j ∈
Revised: ∀ i ∈ [ n ] , ∀ j ∈

Original: [ Ki ]
Revised: [ K

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: 3f)
Revised: 3f

Original: [Equation]
Revised: [Equation]

Original: ∀ t ∈ T , ∀ l ∈ L t .
Revised: ∀ t ∈ T , ∀ l ∈ L t .

Original: (3g)
Revised: (3g)

Original: Eq. (3a) guarantees exactly one active leaf l in leaf set L t for tree t .
Revised: Eq. (3a) guarantees exactly one active leaf l in leaf set L t for tree t .

Original: Eqs.
Revised: Eqs.

Original: (3b)–(3c) ensure that binaryvariables z t,l are only active if all previous splits ν V ( s ) ,j are active.
Revised: (3b)–(3c) ensure that binaryvariables z t,l are only active if all previous splits ν V ( s ) ,j are active.

Original: At any node in a given tree, left ( s )and right ( s ) contain all leaves following the left and right branches, respectively.
Revised: At any node in a given tree, left ( s )and right ( s ) contain all leaves following the left and right branches, respectively.

Original: The mapping V ( s )gives the splitting feature at node s ∈ splits ( t ) in tree t .
Revised: The mapping V ( s )gives the splitting feature at node s ∈ splits ( t ) in tree t .

Original: Tree ensembles can handle continuous,integer and categorical data.
Revised: Tree ensembles can handle continuous,integer and categorical data.

Original: Continuous splits s are defined by x V ( s ) ≤ v V ( s ) ,j conditions, wherev V ( s ) ,j is the learned splitting threshold.
Revised: Continuous splits s are defined by x V ( s ) ≤ v V ( s ) ,j conditions, wherev V ( s ) ,j is the learned splitting threshold.

Original: Therefore, C ( s ) only contains a single index j representingthe threshold v V ( s ) ,j .
Revised: Therefore, C ( s ) only contains a single index j representingthe threshold v V ( s ) ,j .

Original: Categorical splits s are characterized by (subsets of) categories available forfeature V ( s ) and define a splitting condition based on the inclusion of x V ( s ) in the category subset atsplit s .
Revised: Categorical splits s are characterized by (subsets of) categories available forfeature V ( s ) and define a splitting condition based on the inclusion of x V ( s ) in the category subset atsplit s .

Original: For categorical splits, C ( s ) includes the category indices comprising the category subset atsplit s .
Revised: For categorical splits, C ( s ) includes the category indices comprising the category subset atsplit s .

Original: Eq.
Revised: Eq.

Original: (3d) ensures that only one category is active per categorical variable i ∈ C .
Revised: (3d) ensures that only one category is active per categorical variable i ∈ C .

Original: Continuoussplitting thresholds of all trees in the ensemble are ordered according to v i, 1 < v i, 2 < ... <
Revised: Continuoussplitting thresholds of all trees in the ensemble are ordered according to v i, 1 < v i, 2 < ... <

Original: v i,K iwith K i denoting the index for the last split of continuous feature i ∈ N .
Revised: v i,K iwith K i denoting the index for the last split of continuous feature i ∈ N .

Original: To enforce this order,
Revised: To enforce this order,

Original: Eq.
Revised: Eq.

Original: (3e) ensures that binary variables
Revised: (3e) ensures that binary variables

Original: ν
Revised: ν

Original: i,j , corresponding to the split thresholds v i,j , are activatedsequentially.
Revised: i,j , corresponding to the split thresholds v i,j , are activatedsequentially.

Original: The model comprising Eqs.
Revised: The model comprising Eqs.

Original: (3) has no direct dependency on x and is fully definedby binary variables indicating which splits and leaves of the tree model active.
Revised: (3) has no direct dependency on x and is fully definedby binary variables indicating which splits and leaves of the tree model active.

Original: However, to allowthe user to include extra equality and inequality constraints on the input vector x , we bound thecontinuous variables based on the active splits by adding linking constraints [39]:
Revised: However, to allowthe user to include extra equality and inequality constraints on the input vector x , we bound thecontinuous variables based on the active splits by adding linking constraints [33]:

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: = { j ∈
Revised: = { j ∈

Original: [ K
Revised: [ K

Original: [Equation]
Revised: [Equation]

Original: ] |
Revised: ] |

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: For categorical variables, Eq.
Revised: For categorical variables, Eq.

Original: (4d) maps the indices of active categories onto x i for i ∈ C .
Revised: (4d) maps the indices of active categories onto x i for i ∈ C .

Original: Theseoptimization formulations are implemented in open-source software ENTMOOT [53] and OMLT [7].
Revised: Theseoptimization formulations are implemented in open-source software ENTMOOT [43] and OMLT [7].

Original:  Tree ensemble kernels for Bayesian optimization
Revised:  Tree ensemble kernels for Bayesian optimization

Original: The technical details in Section 3 are insufficient to use the tree ensemble kernel in a Bayesianoptimization framework.
Revised: The technical details in Section 3 are insufficient to use the tree ensemble kernel in a Bayesianoptimization framework.

Original: While Eqs.
Revised: While Eqs.

Original: 3 and 4 allow optimization over the mean of an associatedacquisition function [38], Bayesian optimization also requires quantifying model uncertainty forexploration.
Revised: 3 and 4 allow optimization over the mean of an associatedacquisition function [32], Bayesian optimization also requires quantifying model uncertainty forexploration.

Original: This section proposes a mixed-integer second-order cone optimization formulationto capture the standard deviation of a GP with a tree kernel prior.
Revised: This section proposes a mixed-integer second-order cone optimization formulationto capture the standard deviation of a GP with a tree kernel prior.

Original: Combining this optimization formulation with the already-developed mixed-integer formulation of the mean function, we derivethe lower confidence bound (LCB) of the tree kernel-based GP.
Revised: Combining this optimization formulation with the already-developed mixed-integer formulation of the mean function, we derivethe lower confidence bound (LCB) of the tree kernel-based GP.

Original: The advantage of deriving a mixedinteger second-order cone optimization formulation of the LCB acquisition function is that we canglobally optimize the acquisition function.
Revised: The advantage of deriving a mixedinteger second-order cone optimization formulation of the LCB acquisition function is that we canglobally optimize the acquisition function.

Original: Additionally, we can easily incorporate explicit inputconstraints that capture domain knowledge and/or known search space relationships.
Revised: Additionally, we can easily incorporate explicit inputconstraints that capture domain knowledge and/or known search space relationships.

Original: Our optimization problem, which includes the LCB acquisition function, is:
Revised: Our optimization problem, which includes the LCB acquisition function, is:

Original: [Equation]
Revised: [Equation]

Original: with µ ( x ) and σ ( x ) denoting the surrogate model’s mean prediction and standard deviation, respectively.
Revised: with µ ( x ) and σ ( x ) denoting the surrogate model’s mean prediction and standard deviation, respectively.

Original: Functions h
Revised: Functions h

Original: ( x ) and g ( x ) are known constraints and handled similarly to [5, 55].
Revised: ( x ) and g ( x ) are known constraints and handled similarly to [5, 45].

Original: In ourimplementation, the constraints can be linear, quadratic, or polynomial.
Revised: In ourimplementation, the constraints can be linear, quadratic, or polynomial.

Original: Hyperparameter κ controlsthe exploitation-exploration trade-off to determine the next black-box function query area.
Revised: Hyperparameter κ controlsthe exploitation-exploration trade-off to determine the next black-box function query area.

Original: Thesolution to Eq.
Revised: Thesolution to Eq.

Original: 5a is defined by x ∗ lb and x ∗ ub for non-categorical variables i ∈ N , i.e. continuousand integer features, and x ∗ cat , a set of valid category subsets for categorical variables i ∈ C .
Revised: 5a is defined by x ∗ lb and x ∗ ub for non-categorical variables i ∈ N , i.e. continuousand integer features, and x ∗ cat , a set of valid category subsets for categorical variables i ∈ C .

Original: Twovectors ( x ∗ lb and x ∗ ub ) define the non-categorical variables because the trees are piecewise-constantover intervals and the vectors define the lower and upper bounds of these intervals.
Revised: Twovectors ( x ∗ lb and x ∗ ub ) define the non-categorical variables because the trees are piecewise-constantover intervals and the vectors define the lower and upper bounds of these intervals.

Original: Next, we formalize the mean and variance of the tree kernel-based GP.
Revised: Next, we formalize the mean and variance of the tree kernel-based GP.

Original: Eqs.
Revised: Eqs.

Original: 6 and 2 are equivalent.
Revised: 6 and 2 are equivalent.

Original: [Equation]
Revised: [Equation]

Original: The Gram matrix K X , X and target vector y are constants in the optimization model as these quantitiesonly depend on the data set X .
Revised: The Gram matrix K X , X and target vector y are constants in the optimization model as these quantitiesonly depend on the data set X .

Original: The value K x , x is directly related to signal variance hyperparameterσ 0 since there is full leaf overlap for two identical inputs:
Revised: The value K x , x is directly related to signal variance hyperparameterσ 0 since there is full leaf overlap for two identical inputs:

Original: [Equation]
Revised: [Equation]

Original: The vector K x , X describes the kernel output of x with individual data points x i .
Revised: The vector K x , X describes the kernel output of x with individual data points x i .

Original: We computethe constant matrix A ∈ R m ×|L| after the gradient-boosted tree is trained but before solving theacquisition function.
Revised: We compute constantmatrix A ∈ R m ×|L| after the gradient-boosted tree is trained but before solving the acquisitionfunction.

Original: The entries of A are equal to 1 for all active leaves l of data point i and 0otherwise.
Revised: The entries of A are equal to 1 for all active leaves l of data point i and 0 otherwise.

Original: Given matrix A , Eqs.
Revised: Givenmatrix A , Eqs.

Original: (8) capture the kernel output by summing up all binary variables z t,lthat are active for data point i .
Revised: (8) capture the kernel output by summing up all binary variables z t,l that are activefor data point i .

Original: [Equation]
Revised: [Equation]

Original: To get an intuition for Eqs.
Revised: To get an intuition for Eqs.

Original: (8), note that the largest possible value for each k Tree ( x , x i ) is σ 20 (if thecorresponding entries of matrix
Revised: (8), note that the largest possible value for each k Tree ( x , x i ) is σ 20 (if thecorresponding entries of matrix

Original: A are all equal to 1 and there is full leaf overlap) and the smallestpossible value for each k Tree ( x , x i ) is 0 (if there is no leaf overlap).
Revised: A are all equal to 1 and there is full leaf overlap) and the smallestpossible value for each k Tree ( x , x i ) is 0 (if there is no leaf overlap).

Original: In general, values of each elementof vector K x , X will range between 0 and σ 20 : higher values in the elements of K x , X indicate a higherdegree of overlap between the next query location x and data point x i in the set X .
Revised: In general, values of each elementof vector K x , X will range between 0 and σ 20 : higher values in the elements of K x , X indicate a higherdegree of overlap between the next query location x and data point x i in the set X .

Original: Without considering additional tree model constraints h and g , the resulting acquisition function is amixed-integer quadratic optimization problem.
Revised: Without considering additional tree model constraints h and g , the resulting acquisition function is amixed-integer quadratic optimization problem.

Original: The optimization problem is mixed-integer becauseof binary variables ν and quadratic because of Eq.
Revised: The optimization problem is mixed-integer becauseof binary variables ν and quadratic because of Eq.

Original: (6b).
Revised: (6b).

Original: The quadratic Eq.
Revised: The quadratic Eq.

Original: (6b) components areσ 2 and the terms k 2 Tree ( x , x i ) arising from the inner product of K x , X with itself.
Revised: (6b) components areσ 2 and the terms k 2 Tree ( x , x i ) arising from the inner product of K x , X with itself.

Original: We only requireone direction of the Eq.
Revised: We only requireone direction of the Eq.

Original: (6b) equality ( ≤ ) and re-write Eq.
Revised: (6b) equality ( ≤ ) and re-write Eq.

Original: (6b) as a second-order cone constraint.
Revised: (6b) as a second-order cone constraint.

Original: Solver Gurobi 9 [21] automatically finds that Eq.
Revised: Solver Gurobi 9 [19] automatically finds that Eq.

Original: (6b) (with ≤ rather than = ) can be represented as asecond-order cone and makes the appropriate algorithm modifications, e.g. as described by [26, 57].
Revised: (6b) (with ≤ rather than = ) can be represented as asecond-order cone and makes the appropriate algorithm modifications, e.g. as described by [24, 46].

Original: The acquisition function, with Objective 5a and Constraints (3)–(8b), is a mixed-integer second-ordercone program which can be solved with optimization solvers.
Revised: The acquisition function, with Objective 5a and Constraints (3)–(8b), is a mixed-integer second-ordercone program which can be solved with optimization solvers.

Original: A valid solution to the proposed modelis a set of active leaves L ∗ and the intersection of all leaf subspaces [ x ∗ lb , x ∗ ub ] , x ∗ cat .
Revised: A valid solution to the proposed modelis a set of active leaves L ∗ and the intersection of all leaf subspaces [ x ∗ lb , x ∗ ub ] , x ∗ cat .

Original: To derive the next black-box function query point x ∗ , we propose some heuristics.
Revised: To derive the next black-box function query point x ∗ , we propose some heuristics.

Original: The acquisitionfunction value is constant for x ∈ [ x ∗ lb , x ∗ ub ] , x ∗ cat , i.e. all contained points are equivalent from the perspective of the tree kernel-GP.
Revised: The acquisitionfunction value is constant for x ∈ [ x ∗ lb , x ∗ ub ] , x ∗ cat , i.e. all contained points are equivalent from the perspective of the tree kernel-GP.

Original: For the continuous and integer features, we observe that tree modelstend to learn split thresholds close to training data points.
Revised: For the continuous and integer features, we observe that tree modelstend to learn split thresholds close to training data points.

Original: We propose the center of [ x ∗ lb , x ∗ ub ] as thenext query point x ∗ for continuous and integer features:
Revised: We propose the center of [ x ∗ lb , x ∗ ub ] as thenext query point x ∗ for continuous and integer features:

Original: [Equation]
Revised: [Equation]

Original: For integer features with a fractional mid-point between the upper and lower bound, we randomlyselect its floor or ceiling.
Revised: For integer features with a fractional mid-point between the upper and lower bound, we randomlyselect its floor or ceiling.

Original: For categorical features, we sample from the subset of available categories:
Revised: For categorical features, we sample from the subset of available categories:

Original: [Equation]
Revised: [Equation]

Original: For the unconstrained case, we use x ∗ mid as the next query point x ∗ .
Revised: For the unconstrained case, we use x ∗ mid as the next query point x ∗ .

Original: When additional input constraintsh ( x ) and/or g ( x ) are given, we know that at least one point in [ x ∗ lb , x ∗ ub ] , x ∗ cat is feasible even if theheuristic x ∗ mid is infeasible.
Revised: When additional input constraintsh ( x ) and/or g ( x ) are given, we know that at least one point in [ x ∗ lb , x ∗ ub ] , x ∗ cat is feasible even if theheuristic x ∗ mid is infeasible.

Original: To repair the solution x ∗ mid , we project it onto the feasible space:x ∗ ∈ arg min
Revised: To repair the solution x ∗ mid , we project it onto the feasible space:x ∗ ∈ arg min

Original: [Equation]
Revised: [Equation]

Original: g
Revised: g

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Eq.
Revised: Eq.

Original: (11) projects x ∗ mid onto the feasible set defined by h ( x ) and g ( x ) .
Revised: (11) projects x ∗ mid onto the feasible set defined by h ( x ) and g ( x ) .

Original: The time complexity ofsolving Eq.
Revised: The time complexity ofsolving Eq.

Original: 11, and that of solving Eqs.
Revised: 11, and that of solving Eqs.

Original: (3)–(6) for the search space [ x ∗ lb , x ∗ ub ] , x ∗ cat , are both NPhard.
Revised: (3)–(6) for the search space [ x ∗ lb , x ∗ ub ] , x ∗ cat , are both NPhard.

Original: However, we found that Gurobi 9 often solves both problems to ϵ -global optimality formoderately-sized tree models in less time compared to random sampling.
Revised: However, we found that Gurobi 9 often solves both problems to ϵ -global optimality formoderately-sized tree models in less time compared to random sampling.

Original: Hyperparameters.
Revised: Hyperparameters.

Original: We train the kernel hyperparameters signal variance σ 20 and noise σ 2 y by maximizing the log marginal likelihood.
Revised: We train the kernel hyperparameters signal variance σ 20 and noise σ 2 y by maximizing the log marginal likelihood.

Original: Additional hyperparameters are introduced by the gradient-boostedtree ensemble, i.e. maximum tree depth and number of trees, and through κ in the LCB.
Revised: Additional hyperparameters are introduced by the gradient-boostedtree ensemble, i.e. maximum tree depth and number of trees, and through κ in the LCB.

Original: For the
Revised: For the

Original: Section 5 numerical studies, we leave κ and gradient-boosting hyperparameters constant and only increase maximum tree depth and number of trees for the high-dimensional CIFAR-NAS benchmark tocapture more complicated interactions.
Revised: Section 5 numerical studies, we leave κ and gradient-boosting hyperparameters constant and only increase maximum tree depth and number of trees for the high-dimensional CIFAR NAS benchmark tocapture more complicated interactions.

Original: The appendix reports specific values for all hyperparameters.
Revised: The appendix reports specific values for all hyperparameters.

Original: Limitations.
Revised: Limitations.

Original: The method suffers from standard BO challenges where the tree kernel may not bea good prior for the underlying black-box function, e.g. purely continuous feature spaces.
Revised: The method suffers from standard BO challenges where the tree kernel may not bea good prior for the underlying black-box function, e.g. purely continuous feature spaces.

Original: Morelimitations arise from solving an NP-hard problem to global optimality when working with large datasets in high-dimensional search spaces.
Revised: Morelimitations arise from solving an NP-hard problem to global optimality when working with large datasets in high-dimensional search spaces.

Original: For cases where solving the NP-hard problem is too difficultbut BO is still applicable, Gurobi 9 can typically develop good feasible solutions as a heuristic.
Revised: For cases where solving the NP-hard problem is too difficultbut BO is still applicable, Gurobi 9 can typically develop good feasible solutions as a heuristic.

Original:  Numerical studies
Revised:  Numerical studies

Original: This section empirically evaluates the performance of tree kernel-based GPs using a wide varietyof synthetic and real-world benchmark problems.
Revised: This section empirically evaluates the performance of tree kernel-based GPs using a wide varietyof synthetic and real-world benchmark problems.

Original: We show (i) the ability of tree kernels to captureuncertainty of the underlying tree ensemble, (ii) the advantage of using global vs. local strategies foroptimizing the acquisition function and (iii) the proposed algorithm’s superior performance in caseswith constrained search spaces and mixed variable types.
Revised: We show (i) the ability of tree kernels to captureuncertainty of the underlying tree ensemble, (ii) the advantage of using global vs. local strategies foroptimizing the acquisition function and (iii) the proposed algorithm’s superior performance in caseswith constrained search spaces and mixed variable types.

Original: LEAF-GP denotes the Section 4 proposedalgorithm, and Section 2 outlines the baseline of methods we compare against.
Revised: LEAF-GP denotes the Section 4 proposedalgorithm, and Section 2 outlines the baseline of methods we compare against.

Original: For every run, wevisualize the median and confidence intervals of the first and third quartile based on 20 individualruns with varying random seeds.
Revised: For every run, wevisualize the median and confidence intervals of the first and third quartile based on 20 individualruns with varying random seeds.

Original: Further technical details can be found in the appendix.
Revised: Further technical details can be found in the appendix.

Original: 5.1 Uncertainty metric
Revised: 5.1 Uncertainty metric

Original: The tree kernel-based GP uses the leaf overlap measure to quantify correlation between two inputs.
Revised: The tree kernel-based GP uses the leaf overlap measure to quantify correlation between two inputs.

Original: In BO, such measures help identify unexplored areas where correlation to existing training data islow and we expect inaccurate model predictions.
Revised: In BO, such measures help identify unexplored areas where correlation to existing training data islow and we expect inaccurate model predictions.

Original: To empirically test the tree kernel’s capability ofidentifying uncertainty in the underlying tree ensemble, we change the optimization formulation:
Revised: To empirically test the tree kernel’s capability ofidentifying uncertainty in the underlying tree ensemble, we change the optimization formulation:

Original: [Equation]
Revised: [Equation]

Original: , x ∗ cat ∈ arg min x , z ,ν µ ( x ) , 
Revised: , x ∗ cat ∈ arg min x , z ,ν µ ( x ) , (12a) s.t. Eq. (3), Eq. (4), Eq. (6), Eq.

Original: [Equation]
Revised: (7), Eq. (8) (12b) |T | − 1 (cid:88) t ∈T (cid:88) l ∈L t A i,l

Original: [Equation]
Revised: [Equation]

Revised: t,l

Original: [Equation]
Revised: [Equation]

Original: 12b)
Revised: R,

Original: [Equation]
Revised: [Equation]

Revised: [ m ] (12c)

Original: 0.9 Tree Agreement Ratio
Revised: 0.9 Tree Agreement Ratio

Original: R e l a t i v e P r e d ic t i o n
Revised: R e l a t i v e P r e d i

Original:  E rr o r (a) Relative Model Error 0.
Revised: c t i o n E rr o r (a) Relative Model Error 0.

Original: 0.9 Tree Agreement Ratio 2.
Revised: 0.9 Tree Agreement Ratio 2.

Original: P r e d i c t i o n
Revised: P r e d i c t i o n

Original: M e a n
Revised: M e a n

Original: [Figure]
Revised: [Figure]

Original: Eq. (12a) minimizes over the mean prediction of the tree kernel-based GP with Eq.
Revised: Eq. (12a) minimizes over the mean prediction of the tree kernel-based GP with Eq.

Original: (12c) restrictingthe ratio of tree agreement for the black-box query point x ∗ defined as the Eq.
Revised: (12c) restrictingthe ratio of tree agreement for the black-box query point x ∗ defined as the Eq.

Original: (9) center of theoptimal area x ∗ lb , x ∗ ub , x ∗ cat .
Revised: (9) center of theoptimal area x ∗ lb , x ∗ ub , x ∗ cat .

Original: A solution to problem Eq.
Revised: A solution to problem Eq.

Original: (12) guarantees a maximum leaf overlap of
Revised: (12) guarantees a maximum leaf overlap of

Original: R between x ∗ and all available data points X .
Revised: R between x ∗ and all available data points X .

Original: Here, we evaluate values of R ∈ [0 . 35 , 1 . 0] withincrements of 0 . 05 and compute the model error according to:
Revised: Here, we evaluate values of R ∈ [0 . 35 , 1 . 0] withincrements of 0 . 05 and compute the model error according to:

Original: [Equation]
Revised: [Equation]

Original: Fig.
Revised: Fig.

Original: 2 shows results for the Rastrigin [51] benchmark.
Revised: 2 shows results for the Rastrigin [42] benchmark.

Original: With smaller values of R , the Eq.
Revised: With smaller values of R , the Eq.

Original: (12)optimization problem becomes more restrictive, leading to larger objective values for solution x ∗ .
Revised: (12)optimization problem becomes more restrictive, leading to larger objective values for solution x ∗ .

Original: Fig. 2(a) shows that a growing leaf overlap, i.e. larger R , reduces the model error defined in Eq.
Revised: Fig. 2(a) shows that a growing leaf overlap, i.e. larger R , reduces the model error defined in Eq.

Original: This suggests that the kernel works as intended and that low kernel correlation can reveal searchspace areas with high model uncertainty.
Revised: This suggests that the kernel works as intended and that low kernel correlation can reveal searchspace areas with high model uncertainty.

Original: The appendix provides results for additional benchmarks.
Revised: The appendix provides results for additional benchmarks.

Original: 5.2 Local vs. global acquisition optimization
Revised: 5.2 Local vs. global acquisition optimization

Original: [Figure]
Revised: [Figure]

Original:  This section (i) compares LEAF-GP to other state-of-the-art algorithms for common benchmarks and(ii)
Revised: c t i v [Figure] This section (i) compares LEAF-GP to other state-of-the-art algorithms for common benchmarks and(ii)

Original: shows the advantage of global vs. local strategies for optimizing the acquisition function.
Revised: shows the advantage of global vs. local strategies for optimizing the acquisition function.

Original: Weintroduce a variation of the proposed algorithm, LEAF-GP-RND , which optimizes the same acquisitionfunction as LEAF-GP , but uses random sampling instead of Gurobi 9.
Revised: Weintroduce a variation of the proposed algorithm, LEAF-GP-RND , which optimizes the same acquisitionfunction as LEAF-GP , but uses random sampling instead of Gurobi 9.

Original: Fig.
Revised: Fig.

Original: 3 shows results for
Revised: 3 shows results for

Original: Hartmann (6D) [51] and Styblinski-Tang (10D) [51].
Revised: Hartmann (6D) [42] and Styblinski-Tang (10D) [42].

Original: We do not expect tree model-based approachesto perform well since both benchmark functions are continuous.
Revised: We do not expect tree model-based approachesto perform well since both benchmark functions are continuous.

Original: However, the benchmarks show thatall approaches perform similarly for Hartmann (6D) with UCB-MATERNand EI-MATERN improving theblack-box objective at the fastest rate.
Revised: However, the benchmarks showthat all approaches perform similarly for Hartmann (6D) with UCB-MATERN improving the black-boxobjective at the fastest rate.

Original: For Styblinski-Tang (10D), LEAF-GP significantly outperformsother algorithms.
Revised: For Styblinski-Tang (10D), LEAF-GP significantly outperforms otheralgorithms.

Original: Although LEAF-GP is not specialized to this setting, observe that it performsroughly with the state-of-the-art on these continuous, unconstrained optimization problems.
Revised: Although LEAF-GP is not specialized to this setting, observe that it performs roughlywith the state-of-the-art on these continuous, unconstrained optimization problems.

Original: Although sampling-based LEAF-GP-RND and optimization-based LEAF-GP perform similarly on thesmaller Hartmann (6D), LEAF-GP is particularly strong on the higher-dimensional Styblinski-Tang(10D) benchmark function, and two additional benchmarks in the appendix.
Revised: Although sampling-based LEAF-GP-RND and optimization-based LEAF-GP perform similarly on thesmaller Hartmann (6D), LEAF-GP is particularly strong on the higher-dimensional Styblinski-Tang(10D) benchmark function, and two additional benchmarks in the appendix.

Original: Moreover, LEAF-GP-RNDdoes not support explicit input constraints and performs particularly bad for BO with known con- straints.
Revised: Moreover, LEAF-GP-RND does not support explicit input constraints and performs particularly bad for BO with known constraints.

Original: For the following constrained benchmarks, we remove LEAF-GP-RND from the comparison.
Revised: For the following constrained benchmarks [Figure] we remove LEAF-GP-RND from the comparison.

Original: The appendix has additional details and results.[Figure]
Revised: The appendix has additional details and results.

Original:  Constrained spaces
Revised: 5.3 Constrained spaces

Original: [Figure]
Revised: [Figure]

Original: median line and confidence intervals (first and third quartile) from 20 random seeds. Confidence intervals are neglected for methods that cannot improve the initial training data.
Revised: c

Original: Figure subtitles give the function name and number of: dimensions (D), equality constraints (EC), and inequality constraints (IC).
Revised: [Figure] c t i v [Figure]

Original: Section 5.3 provides more details.
Revised: Section 5.3 provides more details.

Original: This section presents numerical benchmarks with known input constraints.
Revised: This section presents numerical benchmarks with known input constraints.

Original: The acquisition optimization strategy of LEAF-GP allows explicit consideration of input constraints, i.e. logical andconvex/non-convex n -th degree polynomial equality and inequality constraints for mixed variablespaces.
Revised: The acquisition optimization strategy of LEAF-GP allows explicit consideration of input constraints, i.e. logical andconvex/non-convex n -th degree polynomial equality and inequality constraints for mixed variablespaces.

Original: The UCB-MATERN and EI-MATERN implementations of BoTorch supports only linear equalityand inequality constraints at the acquisition optimization step.
Revised: The UCB-MATERN implementation of BoTorch supports only linear equality and inequalityconstraints at the acquisition optimization step.

Original: The GA pymoo algorithm has aninterface for callable constraint functions which are considered when generating new generations ofcandidate points.
Revised: The GA pymoo algorithm has an interface for callableconstraint functions which are considered when generating new generations of candidate points.

Original: When a method does not support the specific input constraints, we penalize theobjective:
Revised: When a method does not support the specific input constraints, we penalize the objective:

Original: [Equation]
Revised: [Equation]

Original: where g ( x ) ≤ 0 and h ( x ) = 0 are inequality and equality constraints, respectively.
Revised: where g ( x ) ≤ 0 and h ( x ) = 0 are inequality and equality constraints, respectively.

Original: This penaltystrategy allows methods that do not support explicit constraints to still produce feasible points givenenough iterations.
Revised: This penaltystrategy allows methods that do not support explicit constraints to still produce feasible points givenenough iterations.

Original: Eq.
Revised: Eq.

Original: (14) introduces the hyperparameter λ which weights the penalty, we test valuesλ ∈ { 1 , 10 , 100 } and only plot the best run for all methods that rely on constraint penalization.
Revised: (14) introduces the hyperparameter λ which weights the penalty, we test valuesλ ∈ { 1 , 10 , 100 } and only plot the best run for all methods that rely on constraint penalization.

Original: Toinitialize every method with feasible points, we draw random samples from a uniform distributionwithin the bounds that define the search space and compute the closest feasible point similar to
Revised: Toinitialize every method with feasible points, we draw random samples from a uniform distributionwithin the bounds that define the search space and compute the closest feasible point similar to

Original: Eq.
Revised: Eq.

Original: For constraint benchmarks, we introduce FEAS-RANDOM which simply projects randomsamples onto the set of feasible points that satisfy h ( x ) and g ( x ) .
Revised: For constraint benchmarks, we introduce FEAS-RANDOM which simply projects randomsamples onto the set of feasible points that satisfy h ( x ) and g ( x ) .

Original: Fig.
Revised: Fig.

Original: 4 plots feasible solutionsto four different continuous benchmark problems, G1, G3, G4 [25], and Alkylation [48].
Revised: 4 plots feasible solutionsto four different continuous benchmark problems, G1, G3, G4 [23], and Alkylation [40]. The

Original: The G1benchmark has linear inequality constraints only, which are supported by UCB-MATERN , EI-MATERNand LEAF-GP .
Revised: G1 benchmark has linear inequality constraints only, which are supported by UCB-MATERN and LEAF-GP .

Original: Fig.
Revised: Fig.

Original: 4(a) shows LEAF-GP making quick progress at the beginning with UCB-MATERNcatching up towards the end.
Revised: 4(a) shows LEAF-GP making quick progress at the beginning with UCB-MATERNcatching up towards the end.

Original: G3 and G4 have different combinations of nonlinear constraints,and LEAF-GP significantly outperforms competing methods and random feasible sampling.
Revised: G3 and G4 have different combinations of nonlinear constraints,and LEAF-GP significantly outperforms competing methods and random feasible sampling.

Original: The
Revised: The

Original: Alkylation benchmark determines optimal operating conditions for a simplified alkylation processconsidering complicated nonlinear constraints representing economic, physical and performancelimits.
Revised: Alkylation benchmark determines optimal operating conditions for a simplified alkylation processconsidering complicated nonlinear constraints representing economic, physical and performancelimits.

Original: Again, LEAF-GP outperforms other methods, which often struggle to find feasible solutions.
Revised: Again, LEAF-GP outperforms other methods, which often struggle to find feasible solutions.

Original: More details regarding the presented benchmark problems are given in the appendix.
Revised: More details regarding the presented benchmark problems are given in the appendix.

Original: [Figure]
Revised: [Figure]

Original: We now consider spaces that exhibit mixtures of continuous, integer and categorical variables forconstrained problems.
Revised: We now consider spaces that exhibit mixtures of continuous, integer and categorical variables forconstrained problems.

Original: Tree model-based algorithms naturally support categorical variables byreplacing continuous splits with categorical splits.
Revised: Tree model-based algorithms naturally support categorical variables byreplacing continuous splits with categorical splits.

Original: For all methods that do not support categoricalfeatures, we use one-hot encoding.
Revised: For all methods that do not support categoricalfeatures, we use one-hot encoding.

Original: The category with the highest corresponding auxiliary variablevalue is chosen for the subsequent query point.
Revised: The category with the highest corresponding auxiliary variablevalue is chosen for the subsequent query point.

Original: On the Pressure Vessel (4D) [8] benchmark, whichcomprises two continuous features, two integer features and three inequality constraints, Fig.
Revised: On the Pressure Vessel (4D) [8] benchmark, whichcomprises two continuous features, two integer features and three inequality constraints, Fig.

Original: 5(a)shows that LEAF-GP outperforms other black-box optimization methods.
Revised: 5(a)shows that LEAF-GP outperforms other black-box optimization methods.

Original: The CIFAR-NAS (29D) is a high-dimensional benchmark problem with one continuous, 23 integerand five categorical variables.
Revised: The CIFAR NAS (29D) is a high-dimensional benchmark problem with one continuous, 23 integerand five categorical variables.

Original: It describes properties of different layers and training hyperparametersfor a convolutional neural network (CNN) trained on CIFAR-10 [33].
Revised: It describes properties of different layers and training hyperparametersfor a convolutional neural network (CNN) trained on CIFAR-10 [28].

Original: The problem’s search spaceis hierarchical, as different layers, i.e. convolutional and fully-connected layers, can be activated.
Revised: The problem’s search spaceis hierarchical, as different layers, i.e. convolutional and fully-connected layers, can be activated.

Original: Properties describing a layer are only relevant if the layer is active.
Revised: Properties describing a layer are only relevant if the layer is active.

Original: For this problem, we introducetwo types of constraints to guide the LEAF-GP : (i) constraints that allow for feasible network designs,i.e. different values for padding, stride, kernel size and max pooling in earlier layers affects whatkernel size is feasible in following layers and (ii) constraints that ensure that hierarchies are respected.
Revised: For this problem, we introducetwo types of constraints to guide the LEAF-GP : (i) constraints that allow for feasible network designs,i.e. different values for padding, stride, kernel size and max pooling in earlier layers affects whatkernel size is feasible in following layers and (ii) constraints that ensure that hierarchies are respected.

Original: Towards the former, we include constraints to ensure the output size of every convolutional layer ispositive.
Revised: Towards the former, we include constraints to ensure the output size of every convolutional layer ispositive.

Original: In general, tree models are particularly good at capturing hierarchical search space structures.
Revised: In general, tree models are particularly good at capturing hierarchical search space structures.

Original: To enforce hierarchical relationships during the acquisition optimization step, we introduce indicatorconstraints that force layer properties to take predefined default values if the layer is inactive.
Revised: To enforce hierarchical relationships during the acquisition optimization step, we introduce indicatorconstraints that force layer properties to take predefined default values if the layer is inactive.

Original: Thisensures that the optimizer avoids leaves that infer splitting conditions from feature properties ofinactive layers, thus effectively reducing the search space size.
Revised: Thisensures that the optimizer avoids leaves that infer splitting conditions from feature properties ofinactive layers, thus effectively reducing the search space size.

Original: To expedite tests, we train the CNNon half of the CIFAR-10 training data and optimize for test accuracy.
Revised: To expedite tests, we train the CNN onhalf of the CIFAR-10 training data and optimize for test accuracy.

Original: Fig.
Revised: Fig.

Original: 5(b) summarizes theresults of this study and shows that tree model-based algorithms generally outperform UCB-MATERNand EI-MATERN .
Revised: 5(b) summarizes the results ofthis study and shows that tree model-based algorithms generally outperform UCB-MATERN .

Original: Utilizing search space constraints,SMAC and LEAF-GP find the neural architectureswith the highest test accuracy with SMAC slightly outperforming LEAF-GP .
Revised: Utilizingsearch space constraints, LEAF-GP finds the neural architecture with the highest test accuracy.

Original: Since finding feasiblearchitectures for CIFAR-10 is not particularly challenging, Fig. 9 shows a benchmark for tuningvariational autoencoders (VAE) adapted from Daxberger et al. [11] where LEAF-GP significantlyoutperforms other algorithms. Finding feasible VAE architectures is exceptionally difficult given thatonly certain hyperparameter combinations decode the latent encoding back to the original image size.
Revised: Furtherdetails about this benchmark are given in the appendix.

Original:  Conclusion
Revised:  Conclusion

Original: We present a framework for black-box optimization based on tree kernel Gaussian processes thatsimultaneously allows (i) reliable uncertainty quantification of mixed feature spaces and (ii) incorporation of explicit input constraints.
Revised: We present a framework for black-box optimization based on tree kernel Gaussian processes thatsimultaneously allows (i) reliable uncertainty quantification of mixed feature spaces and (ii) incorporation of explicit input constraints.

Original: Although these two needs have been considered separately, weare able to address both simultaneously through the mixed-integer second-order cone formulation of the acquisition function.
Revised: Although these two needs have been considered separately, weare able to address both simultaneously through the mixed-integer second-order cone formulationof the acquisition function.

Original: The numerical studies show that the proposed strategy performs competitively with state-of-the-art algorithms for unconstrained problems andmay significantly outperform existing methods for constrained benchmarks, especially those with mixed feature spaces.
Revised: The numerical studies show that the proposed strategy performs competitively with state-of-the-art algorithms for unconstrained problems and significantly outperformsexisting methods for constrained benchmarks, especially those with mixed feature spaces.

Original: We useoptimization constraints together with the acquisition functions to incorporate domain knowledge and leverage hierarchical search space structures, e g. for neural architecture search.
Revised: We useoptimization constraints together with the acquisition functions to incorporate domain knowledgeand leverage hierarchical search space structures, e.g. for neural architecture search.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes] See Abstract and Section 1.
Revised: [Yes] See Abstract and Section 1.

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] See Section 4.
Revised: [Yes] See Section 4.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [No] There is
Revised: [No] There is

Original: no obvious societal impact of the method proposed.
Revised: no obvious societal impact of the method proposed.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]
Revised: [N/A]3.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] We submit the code to run all experiments for the proposed method.
Revised: [Yes] We submit thecode to run all experiments for the proposed method.

Original: (b) Did you specify all the training details (e g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?

Original: [Yes] See end of Section 4 and supplemental material.
Revised: [Yes] See end of Section 4 and supplemental material.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] We run 20 random seeds for every experiment and report median and confidence intervals, i e. first and third quartile.
Revised: [Yes] We run 20 random seeds for every experiment and reportmedian and confidence intervals, i.e. first and third quartile.

Original: See Section 5.
Revised: See Section 5.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g.,type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] We report the computational
Revised: [Yes] We report the computational

Original: resources used to conduct the experiments in the supplementary material.
Revised: resources used to conduct the experiments in the supplementary material.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] Yes, we cite alllibraries that were used when preparing the numerical studies.
Revised: [Yes] Yes, we cite all libraries that were used when preparing the numerical studies.

Original: For more information,we refer to the supplemental material.
Revised: For more information, we refer to the supplemental material.

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [No] License information for all assets canbe found in the documentation of each of the assets.
Revised: [No] License information for all assets can be found in the documentation of each of the assets.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No]The code to reproduce the paper’s main results will be published open-source after thepeer review process.
Revised: [No] The code to reproduce the paper’s main results will be published open-source after the peer review process.

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: [Table] (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: [Table] (b) Did you describe any potential participant risks, with links to Institutional Review[Table]
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: [Table] (c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?[N/A] A AppendixB General experimental setup
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?

Original: All experimental results presented in Section 5 were evaluated on an HTCondor cluster (see [52]) ofmachines equipped with Intel Core i7-8700 3.20GHz and 16 GB RAM.

Original: Confidence intervals showthe first and third quartile of 20 independent runs with random seeds ∈ [101 , 102 , . .

Original: If notspecifically indicated in Section C, all competing algorithms use default values for all hyperparameters.

Original: To allow for a fair comparison, we give the same set of initial data points to all tested methods forboth constrained and unconstrained benchmark problems.

Original: A set of ten initial samples is used for the

Original: Section 5.4 CIFAR-NAS example due to its high dimensionality.

Original: Five initial points are used for theremaining benchmark problems.

Original: C Algorithms

Original: This section summarizes the different algorithms used for the Section 5 numerical studies.

Original: We giveimplementation details and hyperparameter settings to reproduce the presented results.

Original: C.1 LEAF-GP and LEAF-GP-RND

Original: The LEAF-GP uses LightGBM [30] for training gradient-boosted tree ensembles.

Original: All runs use thehyperparameter value min_data_in_leaf = 1 as the training dataset size needs to be at least twice theminimum number of data points a leaf is based on.

Original: The min_data_in_leaf default value of LightGBMis 20, which would cause run-time errors after initialization.

Original: We also set LightGBM hyperparametermin_data_per_group = 1 .

Original: For the high-dimensional benchmark problem CIFAR-NAS, we setmax_depth = 5 and num_boost_rounds = 100 for training the ensemble in LightGBM, referring tomaximum interaction depth per decision tree and total number of trees in the ensemble, respectively.

Original: For all other benchmarks we use max_depth = 3 and num_boost_rounds = 50 .

Original: We implement the tree ensemble kernel as a non-stationary kernel in GPyTorch [18].

Original: For derivingthe posterior distribution, we use a Gaussian likelihood and standardize the target values of the dataset.

Original: Section 3 introduces signal variance σ 0 and noise term σ y as kernel hyperparameters which arefitted by maximizing the marginal log likelihood over 200 epochs using the Adam solver [31].

Original: Thehyperparameters are constrained by intervals according to σ 0 ∈ [5e − 4 , 0 . 2] and σ y ∈ [0 . 05 , 20 . 0] .

Original: For LEAF-GP the Section 4 acquisition optimization formulation is encoded using gurobipy [21]and solved using Gurobi 9.

Original: Runs are limited to 100 s if the solver finds a feasible solution and arecontinued otherwise.

Original: Moreover, we set heuristics = 0 . 2 and activate the non-convex hyperparameter for benchmarkproblems with non-convex constraints.

Original: LEAF-GP-RND uses a sampling-based strategy that randomlyevaluates the acquisition function at 2000 locations and selects the minimum value.

Original: C.2 GA

Original: We use the standard Genetic Algorithm implementation of the pymoo [4] toolbox for evolutionaryalgorithms and change population_size = 10 given the small evaluation budget.

Original: Default values areused for all other hyperparameters.

Original: C.3 SKOPT-GBRT and SKOPT-RF

Original: We use the default implementation of Scikit-Optimize [24] with random forest and gradient-boostedtrees base estimators for SKOPT-RF and SKOPT-GBRT , respectively.

Original: Default values are used for allhyperparameters.

Original: C.4 SMAC

Original: For SMAC [27] we utilize the most recent Python implementation SMAC3 [36] using random forestmodels.

Original: Moreover, we specify the hyperparameters run_obj =

Original: ‘ quality ’ and activate the deterministic flag to allow for reproducibility.

Original: The Section 5.4 CIFAR-NAS and Section F VAE-NAS benchmarkproblems exhibit hierarchical search space relationships, i.e., hyperparameters of a specific layer areonly relevant if the layer is active.

Original: We use SMAC ’s InCondition function which allows for certainchild features to be considered only if some parent features have certain values, e.g., the stride oflayer n is only considered if number of layers is at least n .

Original: This allows SMAC to capture hierarchicalrelationships explicitly and to avoid evaluating multiple equivalent configurations.

Original: Default values areused for all other hyperparameters.

Original: C.5 UCB-MATERN and EI-MATERN

Original: The UCB-MATERN and EI-MATERN algorithms use the standard upper confidence bound and expectedimprovement implementations of BoTorch [1].

Original: Before fitting a GP, we normalize data features andstandardize data outputs.

Original: The upper confidence acquisition hyperparameter β is set to 1.96.

Original: We negatethe target values and define raw_samples = 200 for the acquisition function maximization.

Original: Forunconstrained cases, the acquisition optimizer uses 100 restarts.

Original: However, for constrained problemswe limit the optimizer to five restarts due to extensive run-times caused by finding feasible solutions.

Original: Default values are used for all other hyperparameters.

Original: D Benchmark problems

Original: D.1 Unconstrained problems

Original: Figure 3 shows the results for Hartmann (6D), Rastrigin (10D), Schwefel (10D) and StyblinskiTang (10D) benchmark functions implemented according to Surjanovic and Bingham [51].

Original: Table 1summarizes number of dimensions and domain evaluated for unconstrained benchmark problems.

Original: [Table]

Original: D.2 Constrained problems

Original: Fig.

Original: 8 presents results of benchmark problems with known constraints.

Original: Domain bounds withoutdecimals indicate integer-valued variable types.

Original: Benchmark examples G1, G3, G4, G6, G7 and

Original: G10 are implemented according to Hedar [25].

Original: The Alkylation benchmark [48] is adapted from anopen-source implementation [50].

Original: To compare methods that do not support specific input constraints,we penalize the black-box function output according to:

Original: [Equation]

Original: where g ( x ) ≤ 0 and h ( x ) = 0 are inequality and equality constraints, respectively.

Original: Minimizing thecombined black-box output allows methods to find feasible solutions.

Original: Eq.

Original: (15) has the hyperparameterλ which weights the penalty, we test values λ ∈ { 1 , 10 , 100 } and only plot the best run for all

Original: methodsthat rely on constraint penalization.

Original: In the tests conducted, LEAF-GP fully supports explicit inputconstraints.

Original: UCB-MATERN and EI-MATERN support linear inequality and equality constraints only.

Original: The evolutionary algorithm GA has built-in constraint consideration but does not guarantee feasiblesolutions.

Original: Other methods, i.e. LEAF-GP-RND , SKOPT-GBRT , SKOPT-RF and SMAC , rely on the Eq.

Original: (15)penalty function.

Original: D.3 CIFAR-NAS

Original: Table 3 gives more details on the Section 5.4 CIFAR-NAS (29D) benchmark problem.

Original: CIFARNAS (29D) has a total of 29 hyperparameters to tune, i.e. 1 continuous, 15 integer, 8 binary and

Original: [Table]

Original: ] , x { 2 , 3 } ∈ [10 . 0 , 200 . ]

Original: 5 categorical variables.

Original: The goal is to select hyperparameter values for a CNN trained in PyTorch[43] on the CIFAR-10 dataset [33] that maximize test accuracy.

Original: The training and test scripts wereadapted from Trencseni [56].

Original: Due to limited computing resources, we train the CNN on half of thetraining data for 10 epochs using the Adam solver [31].

Original: We score networks using the full test data set.

Original: Only certain combinations of stride, padding and filter size for various layers result in feasible neuralarchitectures, e.g. the filter size of one layer may be too large given the output of the previous layer.

Original: Insuch cases the CNN training fails, and the black-box returns the largest black-box value found so far,helping algorithms learn to avoid infeasible neural architectures.

Original: To simplify the training, layer inputsare parameterized based on outputs of the previous layer for convolutional and fully-connected layers,as well as the intermediate connecting layer.

Original: The benchmark introduces categorical variables foractivation function selection.

Original: Methods that do not support categorical features use one-hot encoding.

Original: LEAF-GP has access to constraints capturing feasible neural architectures mainly concerned withthe convolutional layers.

Original: Algorithms can choose to activate at most three convolutional and twofully-connected layers.

Original: To capture constraints for feasible CNNs, we introduce w out ,i as the output ofconvolutional layer i and W out ,i as the layer’s modified output in case max-pooling is applied:

Original: [Equation]

Original: 16a)

Original: [Equation]

Original: 16b)

Original: Convolutional layers use PyTorch’s Conv2D with inputs derived by the optimization algorithms.

Original: PyTorch’s MaxPool2d implements the max-pooling with the commonly-used (2 , 2) kernel size.

Original: [Equation]

Original: (17a

Original: [Equation]

Original: [Equation]

Original: The 32 × 32 image size of CIFAR-10 data defines the input to the full CNN ( W in , 1 ) in Eq.

Original: (17a).

Original: Eq.

Original: (17b) combines filter size F 1 , padding P 1 and stride S 1 of the first convolutional layer tocompute its output size w out , 1 .

Original: Variable W out , 1 captures the final output of the convolutional layerby considering if the layer is activated, i.e., b conv= 1 , and if max-pooling is applied, i.e., b pool= 1 .

Original: Constraints (17c)–(17g) denote the same restrictions for subsequent layers, each using the output sizeof the previous layer as its input size.

Original: Eq.

Original: (17h) ensures that the output of the last convolutional layer

Original: W out , 3 is at least one.

Original: We also enforce that at least one layer be active, which is captured by Eq. (17i).

Original: To break symmetries in the benchmark problems, we introduce Constraints (18a)–(18g):

Original: [Equation]

Original: (18a)

Original: [Equation]

Original: (18b)

Original: [Equation]

Original: (18c

Original: [Equation]

Original: (18d

Original: [Equation]

Original: Eq. (18a) activates layers in a particular order, and Eq.

Original: (18b) deactivates max-pooling when theassociated convolutional layer is inactive.

Original: Constraints (18c)–(18g) set layer-specific hyperparametersto pre-defined default values when the associated layer is inactive.

Original: We select these defaults as thelower bound for non-categorical variables and the first category for categorical variables.

Original: Constraints (19) express the same restrictions for fully-connected layers:

Original: [Equation]

Original: [Table]

Original: pool= x integer [2 , 5]

Original: F 3 = x integer [1 , 3]

Original: S 3 = x integer [0 , 3]

Original: P 3 = x categ.

Original: { ReLU , PReLU , Leaky ReLU }

Original: Act conv= x binary { 0 , 1 } b fc 1 = x integer [2 , 7]

Original: N fc 1 = 2 x categ.

Original: { ReLU , PReLU , Leaky ReLU }

Original: Act fc 1 = x binary { 0 , 1 } b fc 2 = x

Original: Number integer [2 , 7]

Original: N fc 2 = 2 x categ.

Original: { ReLU , PReLU , Leaky ReLU }

Original: Act fc 2 = x

Original: E Additional Results

Original: This section presents additional results supporting the numerical evaluation in Section

Original: [Figure]

Original: [Figure]

Original: R e l a t i v e

Original: P r e d i c t i o n

Original: E rr o r

Original: [Figure]

Original: l a t i v e

Original: [Figure]

Original: r

Original: [Figure]

Original: i

Original: [Figure]

Original: t i o n

Original: [Figure]

Original: rr o r

Original: [Figure]

Original: [Figure]

Original: a c k

Original: [Figure]

Original: x

Original: [Figure]

Original: j e c t i v

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: Table 4 gives more details on the Section 5.4 Variational Autoencoder Neural Architecture Search(VAE-NAS) benchmark problem, which was adapted from Daxberger et al.

Original: As their exactimplementation is not publicly available, we created a benchmark problem based on the paperdescription and training scripts by Rath [46].

Original: VAE-NAS (32D) has a total of 32 hyperparameters totune, i.e., 1 continuous, 20 integer, and 11 categorical variables.

Original: The goal is to select hyperparametervalues for a variational autoencoder (VAE) [32] trained in PyTorch [43] on the MNIST dataset [12]that minimize test loss, i.e., the average loss of when encoding and decoding images from the test set.

Original: We train each tested VAE for 32 epochs using the Adam solver [31] and a fixed batch size of 128.

Original: Only certain combinations of stride, padding, and filter size for various layers result in feasible neuralarchitectures.

Original: The convolutional layer output size W e out is computed as:

Original: [Equation]

Original: Decoder constraints are more complicated given that the size of the VAE output must match theoriginal MNIST image size.

Original: The output of deconvolutional layers is computed according to:

Original: [Equation]

Original: Matching the output and input sizes of the VAE is non-trivial, as hyperparameters of the convolutionaland deconvolutional layers are themselves set by the optimization algorithms.

Original: To simplify thetraining, layer inputs are parameterized based on outputs of the previous layer for convolutional,deconvolutional, and fully-connected layers.

Original: Specifically, fully-connected layers FC2 and FC3 have2 × N lat nodes.

Original: According to Daxberger et al.

Original: [11], we parameterize the size of the last fully-connectedlayer—which can be FC3 , FC4 , or the latent space layer depending on which layers are active—as

Original: C d 1 × 7 × 7 .

Original: This allows methods not supporting explicit input constraints to still easily find a feasibleneural architecture by deactivating all deconvolutional layers and computing an output 16 × 7 × 7 =784 with C d 1 = 16 (the original MNIST image size is 1 × 28 × 28 = 784 ).

Original: This rule supersedes otherlayer size definitions for fully-connected layers.

Original: Moreover, we extend the benchmark by introducingthe nonlinear activation function of each layer as a categorical optimization hyperparameter.

Original: Methodsthat do not support categorical features use one-hot encoding.

Original: The activation function of the lastlayer is fixed as the sigmoid function, superseding other activation function hyperparameters.

Original: The

Original: VAE is trained using the sum of binary cross-entropy loss (reconstruction error) and KL divergence.

Original: We use the same loss function to evaluate the VAE’s performance on the test dataset, giving theblack-box objective in Fig. 9.

Original: To allow for a fair comparison, the same 10 randomly sampled feasiblearchitectures initialize all methods.

Original: LEAF-GP has access to constraints describing feasible neural architectures.

Original: We begin by definingauxiliary variables similar to the benchmark in Section D.3:

Original: [Equation]

Original: w

Original: [Equation]

Original: ∀ i ∈

Original: [Equation]

Original: 22b

Original: [Equation]

Original: ∀ i ∈ [

Original: [Equation]

Original: 22c

Original: [Equation]

Original: ∀ i ∈ [

Original: [Equation]

Original: 22d)

Original: [Equation]

Original: 22e)

Original: [Equation]

Original: 22f) where W ein ,i and W eout ,i denote, respectively, the input and output sizes of convolutional layer i inthe encoder.

Original: We also define an auxiliary variable w eout ,i to track the output size for inactive layers,as well as binary variables b econv ,i corresponding to the active/inactive state of each layer.

Original: Eq.

Original: (22e)and Eq.

Original: (22f) link binary variables

Original: b econv ,i to the number of active convolutional layers in the encoder.

Original: Using these auxiliary variables, the following relations can be expressed:

Original: [Equation]

Original: 23a)

Original: [Equation]

Original: [Equation]

Original: 23e)

Original: Eq. (23a) and Eq. (23b) define the input sizes as the MNIST image input size W ein , 1 = 28 for the firstlayer and the output size of the previous convolution for ensuing layers.

Original: Eq.

Original: (23c) defines the layeri output w eout ,i given the filter size F e i , padding P e i , and stride S e i .

Original: Eq.

Original: (23d) ensures that the actualconvolutional layer output W eout ,i only takes the value of w eout ,i if the layer is active.

Original: Finally, Eq.

Original: (23e)enforces the output size of the encoder to be at least one.

Original: We add similar auxiliary variables and constraints for the deconvolutional layers:

Original: [Equation]

Original: The Eq.

Original: (24g) indicator constraint restricts the decoder output to be the original image size

Original: W dout , 2 = 28 if deconvolutional layers are active.

Original: Another indicator constraint Eq.

Original: (24h) handlesthe aforementioned case where no deconvolutional layer is active and C d 1 = 16 ensures that thedecoder output size can be resized to original MNIST image size.

Original: We emphasize that this rule isintroduced to simplify the feasible architecture search for methods that do not support explicit inputconstraints.

Original: Note that LEAF-GP could add additional constraints to ensure the architecture’s outputsize can always be resized to the original image size of 28 × 28.

Original: [Equation]

Original: Similar to the encoder, Eq.

Original: (25) defines constraints for feasible decoder layers.

Original: For deconvolutionallayers, we also tune output padding O d i .

Original: According to the PyTorch [18] documentation, outputpadding must be smaller than either stride or dilation.

Original: Given that we do not optimize dilation in deconvolutional layers, Eq. (25d) enforces output padding to be smaller than stride.

Original: We introducesimilar constraints for fully-connected layers in both the encoder and decoder:

Original: [Equation]

Original: 26a

Original: [Equation]

Original: 26e)

Original: To break symmetries in the benchmark problem, we add constraints (27a)–(27m):

Original: [Equation]

Original: , ∀ i ∈ [1 , 2] , (27a) ¬ b econv

Original: [Equation]

Original: i → S e i ≤ 1 , ∀ i ∈ [1 , 2] , (27b)

Original: ¬ b econv ,i → P e i ≤ 0 , ∀ i ∈ [1 , 2] , (27c) ¬ b econv ,i → F e i = 2 , ∀ i ∈ [1 , 2] , (27d)

Original: ¬ b econv ,i → Act e i = ReLU , ∀ i ∈

Original: [1 , 2] , (27e) ¬

Original: [Equation]

Original: fc ,i → Act fc i = ReLU , ∀ i ∈ [1 , 4] , (27f)

Original: ¬ b fc , 1 → N fc 1 ≤ 0 , (27g)

Original: ¬ b ddec ,i → C d i ≤ 4 , ∀ i ∈ [1 , 2] , (27h)

Original: ¬ b ddec ,i → S d i ≤ 1 , ∀ i ∈ [1 , 2] , (27i)

Original: ¬ b ddec ,i → P d i ≤ 0 , ∀ i ∈ [1 , 2] , (27j)

Original: ¬ b ddec ,i → O d i ≤ 0 , ∀ i ∈ [1 , 2] , (27k)

Original: ¬ b ddec ,i → F d i = 2 , ∀ i ∈ [1 , 2] , (27l)

Original: ¬ b ddec , 1 → Act d 1 = ReLU (27m)

Original: Constraints (27a)–(27m) set layer-specific hyperparameters to pre-defined default values when theassociated layer is inactive.

Original: We select these defaults as the lower bound for non-categorical variablesand the first category for categorical variables.

Original: While SMAC is unable to handle more complicatedconstraints restricting outputs of deconvolutional layers, it can handle hierarchical search spacestructures.

Original: For VAE-NAS benchmark runs using SMAC as an optimizer we enforce hierarchiesaccording to constraints (27a)–(27m) which deactivate hyperparameters for inactive layers.

Original: The transformation column refers to post-processing computations before passing the hyperparameter value to the neural network training.

Original: The architecture with all layers activated comprises

Original: C1-C2-FC1-FC2-L-FC3-FC4-D1-D2 , with L referring to the latent space layer.

Original: [Table]
Revised: [N/A]

Original: conti.

Original: [ − 4 . 0 , − 2 . 0]

Original: α = 10 x

Original: [Table]

Original: integer [16 , 64]

Original: N lat

Original: = x

Original: [Table]

Original: integer [0 , 2]

Original: N econv

Original: =

Original: x

Original: [Table]

Original: integer [0 , 2]

Original: N efc

Original: = x

Original: [Table]

Original: integer [0 , 2]

Original: N ddec =

Original: x

Original: [Table]

Original: layers integer [0 , 2]

Original: N dfc

Original: =

Original: x

Original: [Table]

Original: integer [2 , 5]

Original: C e 1 = 2 x

Original: [Table]

Original: integer [1 , 2]

Original: S e 1 = x

Original: [Table]

Original: integer [0 , 3]

Original: P e 1 = x

Original: [Table]

Original: categ.

Original: F e 1 = x

Original: [Table]

Original: categ.

Original: { ReLU , PReLU , Leaky ReLU }

Original: Act e 1 = x

Original: [Table]

Original: integer [3 , 6]

Original: C e 2 = 2 x

Original: [Table]

Original: integer [1 , 2]

Original: S e 2 = x

Original: [Table]

Original: integer [0 , 3]

Original: P e 2 = x

Original: [Table]

Original: categ.

Original: F e 2 = x

Original: [Table]

Original: categ.

Original: { ReLU , PReLU , Leaky ReLU }

Original: Act e 2 = x

Original: Fully-connected layer 1 ( FC1 )

Original: Number of nodes integer [0 , 15]

Original: N fc 1 = 64 × x

Original: Activation function categ.

Original: { ReLU , PReLU , Leaky ReLU }

Original: Act fc 1 = x

Original: Fully-connected layer 2 ( FC2 )

Original: Activation function categ.

Original: { ReLU , PReLU , Leaky ReLU }

Original: Act fc 2 = x

Original: Decoder

Original: Fully-connected layer 3 ( FC3 )

Original: Activation function categ.

Original: { ReLU , PReLU , Leaky ReLU }

Original: Act fc 3 = x

Original: Fully-connected layer 4 ( FC4 )

Original: Activation function categ.

Original: { ReLU , PReLU , Leaky ReLU }

Original: Act fc 4 = x

Original: Deconvolutional layer 1 ( D1 )

Original: Number of input channels integer [3 , 6]

Original: C d 1 = 2 x

Original: Stride integer [1 , 2]

Original: S d 1 = x

Original: Padding integer [0 , 3]

Original: P d 1 = x

Original: Output Padding integer [0 , 1]

Original: O d 1 = x

Original: Filter size categ.

Original: F d 1 = x

Original: Activation function categ.

Original: { ReLU , PReLU , Leaky ReLU }

Original: Act d 1 = x

Original: Deconvolutional layer 2 ( D2 )

Original: Number of input channels integer [2 , 5]

Original: C d 2 = 2 x

Original: Stride integer [1 , 2]

Original: S d 2 = x

Original: Padding integer [0 , 3]

Original: P d 2 = x

Original: Output Padding integer [0 , 1]

Original: O d 2 = x

Original: Filter size categ.

Original: F d 2 = x