High-dimensional Additive Gaussian Processes under Monotonicity Constraints
Abstract
We introduce an additive Gaussian process framework accounting for monotonicity constraints and scalable to high dimensions.
Our contributions are threefold.
First, we show that our framework enables to satisfy the constraints everywhere in the input space.
We also show that more general componentwise linear inequality constraints can be handled similarly, such as componentwise convexity.
Second, we propose the additive MaxMod algorithm for sequential dimension reduction.
By sequentially maximizing a squared-norm criterion, MaxMod identifies the active input dimensions and refines the most important ones.
This criterion can be computed explicitly at a linear cost.
Finally, we provide open-source codes for our full framework.
We demonstrate the performance and scalability of the methodology in several synthetic examples with hundreds of dimensions under monotonicity constraints as well as on a real-world flood application.
Introduction
The framework of additive and constrained Gaussian processes.
In high dimension, many statistical regression models are based on additive structures of the form:
[Equation]
Although such structures may lead to more “rigid” models than non-additive ones they result in simple frameworks that easily scale in high dimensions [1, 2].
Generalized additive models (GAMs) [1] and additive Gaussian processes (GPs) [3, 4] are the most common models in a wide range of applications.
The latter can also be seen as a generalization of GAMs that allow uncertainty quantification.
As shown in [3, 4], additive GPs can significantly improve modeling efficiency and have major advantages for interpretability.
Furthermore, in non-additive small-dimensional GP models, adding inequality constraints (in particular monotonicity which will be the main focus of this paper) leads to more realistic uncertainty quantification in learning from real data [5–10].
Contributions.
Our contributions are threefold.
We combine the additive and constrained frameworks to propose an additive constrained GP (cGP) prior.
Our framework is based on a finite-dimensional representation involving one-dimensional knots for each active variable.
The corresponding mode predictor can be computed and posterior realizations can be sampled, both in a scalable way to high dimension.
We suggest the additive MaxMod algorithm for sequential dimension reduction.
At each step, MaxMod either adds a new variable to the model or inserts a knot for an active one.
This choice is made by maximizing the squared-norm modification of the mode function, for which we supply exact expressions with linear complexity.
We provide open-source codes for our full framework.
We demonstrate the performance and scalability of our methodology with numerical examples in dimension up to 1000 as well as in a real-world application in dimension 37.
MaxMod identifies the most important input variables, with data size as low as n = 2 d in dimension d .
It also yields accurate models satisfying the constraints everywhere on the input space.
Range of applicability.
The computational bottleneck of cGPs is sampling from the posterior distribution.
Here, it boils down to sampling a constrained Gaussian vector with dimension equal to the number of knots.
This is done with Hamiltonian Monte Carlo (HMC) [11] which currently works with several hundreds of knots.
Notice that MaxMod enables to minimize this number of knots.
Our framework is illustrated with monotonicity constraints and can be directly applied to other componentwise constraints such as componentwise convexity.
These constraints should be linear and such that satisfying them on the knots is equivalent to satisfying them everywhere (see Section 3.2).
Related literature.
Additive GPs have been considered in [3, 4, 12–14], to name a few.
The benefit of considering inequality constraints in (non-additive) GPs is demonstrated in [5–10, 15] and in application fields such as nuclear safety [5], geostatistics [6], tree distributions [7], econometrics [16], coastal flooding [17], and nuclear physics [18].
Note that there is a bridge between cGPs and (frequentist) splines-based regression with reproducing kernel Hilbert spaces (RKHS) [19, 20].
Our cGP model and MaxMod algorithm are extensions of the works in [5, 6] and [21] (respectively) to the additive case.
Besides GPs, [22] provides a stochastic optimization framework with RKHS that yields regression with inequality constraints.
[23] suggests a non-parametric model for non-negative functions; a problem related to that of monotonicity constraints.
To the best of our knowledge, our framework is the first that enables to satisfy the constraints (especially monotonicity) everywhere and to scale to high dimension (up to one thousand in our experiments).
In particular, the aforementioned cGP works are not applicable in these dimensions.
Furthermore, [22] does not satisfy the constraints everywhere, but only approximately and in expectation, and [23] does not directly address monotonicity constraints.
For instance, with increasing constraints in dimension one, our framework provides posterior sample functions h satisfying h ( u ) ≤ h ( v ) for all u ≤ v .
In contrast, the derivative of h , h ′ ( u ) , would be guaranteed to be non-negative only at some points u in [15].
Also, h ′ ( u ) would only be guaranteed to be above some strictly negative threshold, in expectation in [22].
Note that for unconstrained GP regression, the likelihood and conditional distribution are explicit but their computation cost scales as O ( n 3 ) , where n is the size of the dataset. An active strand of literature then focuses on providing efficient numerical procedures [24, 25]. Because of the constraints, our work is complementary to this strand of literature. Indeed, with cGPs, even if n is small, the numerical challenge of sampling from the posterior distribution remains, as discussed previously.
The main indicator of complexity is thus the number of knots, rather than the dataset size.
The references [24, 25] provide convergence guarantees.
At this stage, we do not provide convergence guarantees in our setting.
To reach that goal, a possibility would be to extend the convergence proof of MaxMod given in [21] from the non-additive to the additive case.
Two comments are in order on this.
First, the proof techniques in [21] are different from these in [24, 25] (and also from standard asymptotic statistics settings, for instance [26]).
Indeed, n is fixed and the number of knots increases in [21], while n increases in [24, 25].
Second, extending the proof of [21] to the additive case is not straightforward, since the various spaces of (non-additive) functions and their closures in [21] would have to be appropriately redefined for additive functions.
Note also that [21] relies on a convergence proof in [27], that would need to be extended from the non-additive to the additive setting.
Paper organization.
Section 2 describes the additive GP framework.
Sections 3 and 4 present our framework for additive cGPs and introduce the MaxMod algorithm.
Section 5 provides the numerical experiments.
Appendix A gathers the technical proofs as well as additional details and experiments.
Framework on additive Gaussian processes
In additive models, GP priors are placed here over the functions y 1 , . . .
, y d in (1) [3, 4].
For i = 1 , . . .
, d , let { Y i ( x i ); x i ∈ [0 , 1] } be a zero-mean GP with covariance function (or kernel)
k i .
Taking Y 1 , . .
, Y d as independent GPs, the process { Y ( x ); x ∈ [0 , 1] d } , with x = ( x 1 , . . .
, x d ) , that results from the addition of Y 1 , . .
, Y d , is also a GP and its kernel is given by
[Equation]
In regression tasks, we often train the GP Y to noisy data ( x κ , y κ ) 1 ≤ κ ≤ n .
We denote x ( κ ) i , for κ = 1 , . . . , n and i = 1 , . . .
, d , the element corresponding to the i -th input of x κ .
By considering additive xxb Y ( x 1 , x 2 ) xxb Y ( x 1 , x 2 ) xxb Y ( x 1 , x 2 )
Gaussian noises ε
[Equation]
N (cid:0) 0 , τ(cid:1) , with ε 1 , · · · , ε n assumed to be independent and independent of
Y , then the conditional process Y n ( x ) :
= Y ( x ) |{ Y ( x 1 ) + ε 1 = y 1 , . . .
, Y ( x n ) + ε n = y n } is GP-distributed with mean function and covariance function given by [3]
[Equation]
where y n =
[ y 1 , . . .
, y n ] ⊤ , k ( x ) = (cid:80) di
=1 k i ( x i ) with k i ( x i ) =
[ k i ( x i , x (1) i ) , . . .
, k i ( x i , x ( n ) i )] ⊤ , and K = (cid:80) di
=1 K i with ( K i ) κ,ℓ
= k i ( x ( κ ) i , x ( ℓ ) i ) for 1 ≤ κ, ℓ ≤ n .
The conditional mean µ and variance v ( · ) =
c ( · , · ) are used for predictions and prediction errors, respectively. 2D illustration.
Figure 1 shows the prediction of an additive GP modeling the function ( x 1 , x 2 ) (cid:55)→ 4( x 1 − 0 .
5) 2 + 2 x 2 .
The GP is trained with a squared exponential (SE) kernel, k
( x , x ′ ) = (cid:80) di
=1 σ
i exp( − ( x i − x ′ i ) 2 / 2 ℓ 2 i ) and with ( x 1 , x 2 ) : (0 . 5 , 0) , (0 . 5 , 0 . 5) , (0 . 5 , 1) , (0 , 0 . 5) , (1 , 0 . 5) .
The covariance parameters θ =
(( σ 21 , ℓ 1 ) , ( σ 22 , ℓ 2 )) , corresponding to the variance and length-scale parameters (respectively), and the noise variance τ 2 are estimated via maximum likelihood [28].
Although the resulting GP does preserve the additive condition, from Figure 1 (left) we can observe that the quality of the prediction will depend on the availability of data.
In our example, we can observe that the GP model does not properly capture the convexity condition along x 1 .
Contributions on additive Gaussian processes under inequality constraints 3.1 Finite-dimensional Gaussian process for fixed subdivisions
In order to satisfy the constraints everywhere (see the next subsection), we introduce the following finite-dimensional GP.
For i = 1 , . . .
, d we consider a one-dimensional subdivision S i (a finite subset of [0 , 1] composed of knots) with at least two knots at 0 and 1 .
Throughout Section 3, S i is fixed, but its data-driven selection is studied in Section 4.
If the number of knots of S i is m i , then the total number of knots is given by m = m 1 + · · · + m d .
We let S =
( S 1 , . . .
, S d ) .
The finite-dimensional
GP, denoted by Y S ( x ) , is written, for x ∈ [0 , 1] d ,
[Equation]
where ξ i,j = Y i ( t ( S i ) ( j ) ) with Y 1 , . . .
, Y d GPs as in Section 2, and with 0 =
t ( S i ) (1) < · · · < t ( S i ) ( m i ) =
the knots in S i .
We let t ( S i ) (0) =
− 1 and t ( S i ) ( m i +1)
= 2 by convention, and ϕ i,j = ϕ
t ( S i ) ( j − 1) ,t ( S i ) ( j ) ,t ( S i ) ( j +1) : [0 , 1] → R is the hat basis function centered at the knot t ( S i )
( j ) of S i .
That is, for u < v < w , we let [Equation] [Equation] for
[Equation]
[Equation]
[Equation]
1.00 x ϕ ( x ) - -    0.
1.00 x
Y ( x ) - -    0. 1.00 x Y ( x ) Observe from (3) that, since ξ i,j , for i = 1 , . . .
, d and j = 1 , . . .
, m i , are Gaussian distributed, then
Y i, S i is a GP with kernel given by
[Equation]
i
[Equation]
Moreover, Y S is a GP with kernel (cid:101)
[Equation]
(cid:80)
[Equation]
(cid:101) k
i ( x i , x ′ i ) .
Figure 2 shows random realizations from a standard GP and a finite-dimensional GP using a SE kernel with θ = ( σ 2 , ℓ ) = (1 , 0 . 1) . We remark that we consider asymetric basis functions here, which is a novelty compared to [5, 6], that will allow MaxMod (Section 4) to promote non-equispaced designs.
The additive finite-dimensional kernel in (5) also seems not to appear in the existing literature.
3.2 Satisfying inequality constraints
We consider the componentwise constraints
Y i, S i ∈
E i , i = 1 , . . .
, d , where E i is a one-dimensional function set.
In line with [5, 6], we assume that there are convex sets
[Equation] such that
[Equation]
where ξ i =
[ ξ i, 1 , · · · , ξ i,m i ] ⊤ .
Furthermore, for i = 1 , . . . , d , C i is itself composed by q i linear inequalities: [Equation]
[Equation]
where the λ ( i ) κ,j ’s encode the linear operations and the l ( i ) κ ’s and u ( i ) κ
’s represent the lower and upper bounds.
We write the constraints in (7) as l i ≤ Λ i c ≤ u i .
Examples of such constraints are monotonicity and componentwise convexity.
For instance, for the case where the function y in (1) is non-decreasing with respect to each input, then E i is the set of non-decreasing functions on [0 , 1] and C i is given by
[Equation]
Hence in this case, Y S is monotonic on the entire [0 , 1] d if and only if (by definition)
each of its additive component Y i, S i is monotonic on [0 , 1] , which happens if and only if the m i − 1 constraints (8) are satisfied. We discuss the example of componentwise convexity in Appendix A.1.
3.3 Constrained GP predictions
We let Σ i
= k i ( S i , S i ) be the m i ×
m i covariance matrix of ξ i .
We consider x 1 , . . .
, x n ∈ [0 , 1] d and write Φ i for the n × m i matrix with element ( a, b ) given by ϕ i,b ( x a ) .
Then
[Equation]
By considering noisy data ( x κ , y κ ) 1 ≤ κ ≤ n , we have the regression conditions Y n + ε n = y n , where ε n ∼ N (0 , τ 2 I n ) and is independent from ξ 1 , . . . , ξ d .
Then given the observations and the constraints, the maximum à posteriori (MAP) estimate, also called the mode function, is given by
[Equation]
The vector
[Equation]
with
[Equation] ξ i,m i ]
⊤ is the mode of the Gaussian distribution
N ( µ c , Σ c ) of the values at the knots conditionally to the observations and truncated from the constraints l 1 ≤ Λ 1 ξ 1 ≤ u 1 , . . .
, l d ≤ Λ d ξ d ≤ u d as in (7):
[Equation]
Above µ c = [ µ ⊤ c, 1 , . . .
, µ ⊤ c,d ] ⊤ is the m × 1 vector with block i given by
[Equation]
and ( Σ c,i,j ) i,j is the m × m matrix with block ( i, j ) given by
[Equation]
The expressions for µ c and Σ c are obtained from the conditional formulas for Gaussian vectors [28].
Note that the matrix Σ c is not block-diagonal, that is one cannot compute the posterior mode along each dimension separately.
In Appendix A.2, based on the matrix inversion lemma [28, 29], efficient implementations to speed-up the computation of µ c and Σ c are given when m ≪ n .
Notice that the covariance parameters θ of the kernel in (5) are estimated via maximum likelihood once the (unconstrained) additive GP framework is established.
This is an intermediate step before solving the optimization problem in (11).
Hence, given µ c , Σ c , Λ 1 , . . .
, Λ d , l 1 , . . .
, l d , u 1 , . . . , u d , and θ , (11) is convex and can be solved via quadratic programming [5, 30].
To the best of our knowledge, the expression of the mode in (11) does not appear in the existing literature.
The cGP mode in (10) can be used as a point estimate of predictions.
Since trajectories of [ ξ ⊤ 1 , . . . , ξ ⊤ d ] ⊤ conditionally on the observations and constraints can be generated via HMC [11], they can be used for uncertainty quantification.
Furthermore, the mean of the HMC samples can be used for prediction purposes.
Continuing the illustrative 2D example in Figure 1, we see the improvement brought by the mode function and the mean of conditional simulations, compared to the unconstrained additive GP model.
Additive MaxMod algorithm 4.1 Squared-norm criterion
Consider an additive cGP model that uses only a subset J ⊆ { 1 , . . .
, d } of active variables, with cardinality |J | .
We write its (vector of) subdivisions as S =
( S i ; i ∈ J ) .
Its mode function (cid:98)
Y S is defined similarly as in (10), from R |J| to R , by, for x = ( x i ; i ∈ J ) ,
[Equation]
Adding a new active variable i ⋆ ̸∈ J to J , and allocating it the base (minimal) subdivision S i ⋆ = { 0 , 1 } defines a new mode function (cid:98)
Y S ,i ⋆ : R |J| +1 → R .
Adding a knot t ∈ [0 , 1] to the subdivision
S i ⋆ for i ⋆ ∈ J also defines a new mode function (cid:98)
Y S ,i ⋆ ,t : R |J| → R .
Since a new variable or knot increases the computational cost of the model (optimization dimension for the mode computation and sampling dimension for generating conditional trajectories via HMC), it is key to quantify its contribution benefit.
We measure this benefit by the squared-norm modification of the cGP mode
[Equation]
where in the first case we see (cid:98)
Y S as a function of |J | +1 variables that does not use variable i ⋆ .
4.2 Analytic expressions of the squared-norm criterion
Adding a variable.
For a new variable i ⋆ ̸∈ J , the new mode function is
[Equation]
where ( (cid:101) ξ i,j ) i,j and ( (cid:101)
ξ i ⋆ ,j ) j follow from the optimization problem in dimension [Equation] i ∈J m i +corresponding to (10) and (11).
We let ϕ i ⋆ , 1 ( u ) = 1 − u and ϕ i ⋆ , 2 ( u ) =
u for u ∈ [0 , 1] .
Note that even though, for i ∈ J and j = 1 , . . .
, m i , [Equation] i [Equation]
j and [Equation] ξ i,j correspond to the same basis function ϕ i,j , they are not equal in general as the new mode is reestimated with two more knots, which can modify the coefficients of all the knots.
Next, we provide an analytic expression of the integral in (13).
Proposition 1 (See proof in Appendix A.3) .
We have
I S ,i ⋆ =
[Equation]
where η i,j =
[Equation]
:= (cid:82) 1 0 ϕ i,j ( t ) dt and
E ( S i ) j,j ′ :
= (cid:82) 1 [-
 φ-]0 ϕ i,j ( t ) ϕ
i,j ′ ( t ) dt with explicit expressions in Lemma 1 in Appendix A.3.
The matrices ( E ( S i ) j,j ′ ) 1 ≤ j,j ′ ≤ m i are 1-band and the computational cost is linear with respect to m =
(cid:80) i ∈J m i .
Inserting a knot to a variable.
For a new t added to S i ⋆ with i ⋆ ∈ J , the new mode function is
[Equation]
where (cid:101) m i = m i for i
̸
= i ⋆ , (cid:101) m i ⋆ = m i ⋆ + 1 , (cid:101) ϕ
i,j = ϕ
i,j for i ̸ = i ⋆ , and (cid:101) ϕ i ⋆ ,j is obtained from S i ⋆ ∪ { t } as in Lemma 1.
As before, this follows from the optimization problem in dimension (cid:80) i ∈J m i + 1 corresponding to (10) and (11).
Next, we provide an analytic expression of (14).
Proposition 2 (See proof in Appendix A.3) .
For i ∈
J \{ i ⋆ } , let (cid:101)
S i =
S i .
Let (cid:101)
S i ⋆ =
S i ⋆ ∪ { t } .
Recall that the knots in S i ⋆ are written
= t ( S i⋆ ) (1) < · · · < t ( S i⋆ ) ( m i⋆ ) = 1 .
Let ν ∈ { 1 , . .
, m i ⋆ − 1 } be such that t ( S i⋆ ) ( ν ) <
t < t ( S i⋆ ) ( ν +1) .
Then, with a linear computational cost with respect to (cid:101) m = (cid:80) i ∈J (cid:101) m
i , we have
I S ,i ⋆ ,t =
[Equation]
S
[Equation]
j (cid:19) , where ¯ η i,j = ¯ ξ i,j − (cid:101) ξ i,j , E ( (cid:101) S i )
j and E ( (cid:101) S i )
j,j ′ are as in Proposition 1, ¯ ξ i,j = (cid:98) ξ i,j for i ̸
= i ⋆ , ¯ ξ i ⋆ ,j = (cid:98) ξ i ⋆ ,j for j ≤ ν , ¯ ξ i ⋆ ,j = (cid:98) ξ i ⋆ ,j − 1 for j ≥ ν + 2 , and ¯ ξ i ⋆ ,ν +1 = (cid:98) ξ i ⋆
[Equation]
t ( S
[Equation]
t
[Equation]
+
[Equation]
4.3 The MaxMod algorithm
Algorithm 1 summarizes the routine of MaxMod.
When considering inserting a knot t to an active variable i , we promote space filling by adding a reward of the form ∆ D ( t, S i ) , where S i is the current subdivision and where D ( t, S i ) is the smallest distance from t to an element of S i .
When adding a new variable, we add a fixed reward ∆ ′ .
Both ∆ and ∆ ′ are tuning parameters of the algorithm and allow to promote adding new variables over refining existing ones with new knots, or conversely.
Step 3 in Algorithm 1 is performed by a grid search, and involves multiple optimizations for computing the new modes in (11), followed by applications of (14). Step 4 yields a single [Equation] [Equation]
[Equation]
[Equation] [Equation] [Equation] optimization for computing the new mode in (11), followed by an application of (13).
For each computation of a new mode, the covariance parameters of the kernels k i , for i ∈ J , are estimated.
For faster implementations, the covariance parameters can be fixed throughout each pass corresponding to a fixed value of m (Steps 2 to 6) or can be re-estimated every T values of m , for some period
T ∈ N .
Furthermore, steps 3 and 4 can be parallelized and computed in different clusters
Numerical experiments
Implementations of the additive cGP framework are based on the R package lineqGPR [31].
Experiments throughout this section are executed on an 11th Gen Intel(R) Core(TM) i5-1145G7 2.60GHz 1.50 GHz, 16 Gb RAM.
Both R codes and notebooks to reproduce some of the numerical results are available in the Github repository: https://github.com/anfelopera/lineqGPR .
As training data, we use random Latin hypercube designs (LHDs).
Note that using them is not necessary for the implementation of our methodology, but it is recommended when the user is able to design the experiments. Indeed, [32] shows that LH sampling reduces estimation variances compared to simple random sampling, especially for additive functions.
Thus, using LHDs enables us to choose a minimal design size n = 2 d , corresponding to the number of parameters of the chosen additive GP kernel: an additive Matérn 5/2 kernel with one variance parameter σ 2 i and one characteristic-length parameter ℓ i per dimension [3].
We denote θ =
(( σ 21 , ℓ 1 ) , · · · , ( σ 2 d , ℓ d )) .
For other applications where n ∈ [ d + 1 , 2 d [ , we may consider the kernel structure in [4] with only one global variance parameter, at the cost of more restricted cGP models.
Finally, in the examples that require a comparison with non-additive GP models, we use common GP settings: maximin LHDs and n = 10 d .
The larger value of n accounts for the fact that additivity is not a prior information of standard GPs.
We compare the quality of the GP predictions in terms of the Q 2 criterion on unobserved data.
It is defined as Q 2 = 1 − SMSE , where SMSE is the standardized mean squared error [28].
For noise-free data, Q 2 is equal to one if predictions are exactly equal to the test data and lower than one otherwise.
5.1 Additive Gaussian processes with monotonicity constraints 5.1.1 Monotonicity in 5D
We start by an example in small dimensions, in order to compare with non-additive constrained GPs which do not scale to high dimensions.
We thus consider the additive function given by
[Equation]
with x ∈ [0 , 1] 5 .
Observe that y is non-decreasing with respect to all its input variables.
We evaluate y on a maximin LHD over [0 , 1] 5 at n = 50 locations using [33].
In this example, as explained in the introduction of the section, we have chosen a maximin LHD rather than a random LHD, and x 1 xy ( x 1 , x 2 , 0 , 0 , 0 ) x 1 xy ( x 1 , 0 , x 3 , 0 , 0 ) x 1 xy ( x 1 , 0 , 0 , x 4 , 0 ) x 1 xy ( x 1 , 0 , 0 , 0 , x 5 ) x 1 xb Y ( x 1 , x 2 , 0 , 0 , 0 ) x 1 xb Y ( x 1 , 0 , x 3 , 0 , 0 ) x 1 xb Y ( x 1 , 0 , 0 , x 4 , 0 ) x 1 xb Y ( x 1 , 0 , 0 , 0 , x 5 )
[Table]
[Figure]
[Table] [Figure] [Table] n = 10 d rather than n = 2 d , because we also consider non-additive GPs for which these settings are recommended.
We fix 20 knots per dimension, leading to a total of m = 100 knots.
Figure 3 shows the additive cGP mean prediction under monotonicity constraints considering
HMC samples.
Our framework leads to improvements on both CPU time and quality of predictions compared to the non-additive cGP in [5].
Due to computational limitations, the non-additive cGP is performed with a reduced but tractable number of knots per dimension set to m 1 = m {+
+} = m
= 4 , m 3 = 2 and m 5 = 6 , for a total of m = 768 knots.
With this setup, and considering a factorial design with 11 test points per dimension, the additive cGP yields Q 2 = 99 .
8% , an absolute improvement of 1.3% compared to the non-additive cGP.
In term of the CPU times, the computation of the cGP mode and cGP mean using the additive framework are obtained in 6s and 0.8s (respectively), a significant speed-up compared to the non-additive cGP that required 28.9s and 24.3 minutes.
5.1.2 Monotonicity in hundreds of dimensions
For testing the additive cGP in high dimensions, we consider the target function used in [21]:
[Equation]
(16) with x ∈ [0 , 1] d .
This function exhibits decreasing growth rates as the index i increases.
For different values of d , we assess GPs with and without constraints.
We fix θ = ( σ 2 i , ℓ i ) 1 ≤ i ≤ d = (1 , 2) .
Although θ can be estimated our focus here is to assess both computational cost and quality of cGP predictors.
The cGP mean is obtained by averaging 10 3 HMC samples.
We set 5 knots per dimension.
1 summarizes the CPU times and Q 2 values of the cGP predictors.
The Q 2 criterion is computed considering 10 5 evaluations of y based on a LHD fixed for all the experiments.
Results are shown over 10 replicates with different random LHDs.
From Table 1, it can be observed that the cGPs lead [Table] to prediction improvements, with Q 2 (median) increments between 1.7-5.8%.
Although the cGP mean predictor provides Q 2 values above 88%, it becomes expensive when d ≥ 500 .
On the other hand, the cGP mode yields a trade-off between computational cost and quality of prediction.
5.2 MaxMod algorithm 5.2.1 Dimension reduction illustration
We test the capability of MaxMod to account for dimension reduction considering the function in (16).
In addition to ( x 1 , . . .
, x d ) , we include D − d virtual variables, indexed as ( x d +1 , . . .
, x D ) , which will compose the subset of inactive dimensions.
For any combination of D ∈ { 10 , 20 } and d ∈ { 2 , 3 , 5 } , we apply Algorithm 1 to find the true d active dimensions.
For each value of D , we evaluate f at a maximin LHD with n = 10 D .
Similarly to Section 5.1.1 where we compare to non-additive GPs, we have used the common GP settings rather than the ones used here for additive
GPs (random LHD, n = 2 d ).
We compare the accuracy of two modes:
• (cid:98) Y MaxMod : the mode resulting from the additive cGP and additive MaxMod.
• (cid:101) Y MaxMod : the mode resulting from the non-additive cGP in [5] with equispaced onedimensional knots but where the number of knots per dimension is the same as for (cid:98) Y MaxMod .
Table 2 shows that MaxMod correctly identifies the d dimensions that are actually active where the most variable ones have been refined with more knots.
There, MaxMod is considered to converge if the squared-norm criterion is smaller than ϵ = 5 × 10 − 4 .
In terms of the Q 2 criterion, (cid:98)
Y MaxMod leads to more accurate results compared to
[Equation] , with Q 2 ≥ 99 .
7% in all the cases.
5.2.2 Real application: flood study of the Vienne river
The database contains a flood study conducted by the French multinational electric utility company
EDF in the Vienne river [34].
It is composed of N = 2 × 10 4 simulations of an output H representing the water level and 37 inputs depending on: a value of flow upstream, data on the geometry of the bed, and Strickler friction coefficients (see further details in Appendix A.5.1).
This database was generated as a random design, assuming that the inputs are independent and follow specific probability distributions.
In order to apply our framework, we use componentwise quantile transformations to make the inputs uniform.
Such transformations do not modify the additivity and monotonicity properties, but may increase non-linearities.
With expert knowledge, it is possible to identify that H is decreasing along the first 24 input dimensions and increasing along dimension 37 .
The behavior along inputs 25 to 36 is not clear.
Although underlying functions modeling the flood phenomena may be non additive, a previous sensitivity analysis in [34], done with the full database, has shown that the additive assumption is realistic here. Moreover, it has shown that inputs 11, 35 and 37 explain most of the variance.
We aim at obtaining similar conclusions with a fraction of the simulation budget, using the information of additivity and monotonocity constraints with respect to dimensions 1 to 24 and 37.
In this experiment, MaxMod is applied for 10 replicates with random training sets and 10 iterations.
The settings described at the beginning of Section 5 are followed as much as possible.
As the database is fixed, we cannot directly use a random LHD of size n = 2 d .
Thus, we pick the closest subset of size n (measured by the Euclidean norm).
The remaining data are used for testing the cGPs.
Figureshows the choice of MaxMod per iteration and for the 10 replicates (adding or refining variables). M a x M o d I t e r a t i o n reps
[Figure]
− l og ( 1 −
Q [Figure] [Figure] The first three iterations of the algorithm show that MaxMod activates mainly dimensions 11, 35 and 37.
In the subsequent iterations, MaxMod either refines the these dimensions or activates dimensions 12 and 36.
In terms of the quality of predictions (Figure 4), MaxMod leads to Q 2 > 0 .
92 after the first three iterations.
Similar results are obtained for n = 3 d and n = 4 d (see Appendix A.5.2).
MaxMod indeed activates dimensions 11, 35 and 37 in the first three iterations and refines them mainly in subsequent iterations.
This allows to conclude that MaxMod correctly identifies the most relevant input dimensions and that accurate predictions are obtained once those dimensions are activated.
Acknowledgments and Disclosure of Funding
This research was conducted within the frame of the ANR GAP Project (ANR-21-CE40-0007) and the consortium in Applied Mathematics CIROQUO, gathering partners in technological research and academia in the development of advanced methods for Computer Experiments (doi:10.5281/zenodo.6581217).
We thank Bertrand Iooss and EDF R&D/LNHE for providing the Mascaret test case and Sébastien Petit who has performed the computations on this model
A Appendix A.1 Satisfying
inequality constraints everywhere for componentwise
convexity
For componentwise convexity, E i in (6) is the set of one-dimensional convex functions and C
i is given by, with ( t 1 , . . .
, t m i ) =
( t ( S i ) (1) , . . .
, t ( S i ) ( m i ) ) ,
[Equation]
We can see that, for each i ∈ { 1 , . . .
, d } and each x − i ∈ [0 , 1] d − 1 , the one-dimensional cut u ∈ [0 , 1] (cid:55)→ Y S ( u, x − i ) (where only the input i is varying) is convex if and only if each additive component
Y i, S i is convex on [0 , 1] .
This happens if and only if the m i − 2 inequality constraints in (17) are satisfied.
A.2 Speed-up of numerical implementation
Notation.
The expression (9) can be rewritten in the matrix form Y n := (cid:80) di
=1 Φ i ξ i =
Ψ ξ with Ψ =
[ Φ 1 , . . . , Φ d ] an n × m matrix and ξ = [ ξ ⊤ 1 , . . . , ξ ⊤ d ] ⊤ an m × 1 vector.
With this notation, we have the following expressions for µ c and Σ c (see Section 3.3):
[Equation]
where C = ΨΣΨ ⊤ + τ 2 I n and Σ = bdiag( Σ 1 , . .
. , Σ d ) is an m × m block diagonal matrix.
The computation of C − 1 , required in GP predictions and estimation of the covariance parameters via maximum likelihood, can be performed more efficiently when m ≪ n using properties of matrices [28, 29].
Next, we detail how the computational complexity of C − 1 can be reduced from
O ( n 3 ) to O ( m 3 ) .
We also provide an efficient computation of the determinant | C | that is required in covariance parameter estimation.
Computation of C − 1 .
To avoid numerical instability issues, we first rewrite C in terms of the
Cholesky decomposition Σ = LL ⊤ .
Here, L is an m × m block lower-triangular matrix given by L = bdiag( L 1 , · · · , L d ) with L i the Cholesky decomposition of Σ i for i = 1 , . . .
, d .
Thus,
C − 1 =
[( Ψ L )
I m ( Ψ L ) ⊤ + τ 2 I n ] − 1 .
Now, by applying the matrix inversion lemma (see, e.g., [28],
Appendix A.3), we obtain:
[Equation]
We need to compute now the inversion of the m × m matrix A
= τ
I m + L ⊤ Ψ ⊤ Ψ L .
Let (cid:101)
L be the
Cholesky decomposition of A .
Denote P = Ψ L .
Then C − 1 is given by
[Equation]
Since (cid:101)
L is a lower triangular matrix, then (cid:101) LM =
P ⊤ can be sequentially solved in M .
In addition to reducing complexity to O ( m 3 ) for m ≪ n , some of the intermediate steps here can be parallelized.
For instance, the computation of the d Cholesky matrices L i with m i < m .
Computation of | C | .
From [28] (Appendix A.3), and considering the Cholesky decomposition Σ = LL ⊤ , we have that the determinant | C | is given by
[Equation]
where |
[Equation]
L j,j with (cid:101)
L j,j the element associated to the j -th row and j -th column of (cid:101)
L .
A.3 Squared-norm criterion
The next lemma is elementary to show and its second part is also given in [21].
Lemma 1. Consider a subdivision S
= { u 1 , . . .
, u m } and write its ordered knots as
= u (1) < · · · < u ( m ) = 1 .
For j = 1 , . . .
, m , write the hat basis function ϕ j as ϕ u ( j − 1) ,u ( j ) ,u ( j +1) in (4) , with the conventions u (0) = − 1 and u ( m +1)
= 2 .
For j = 1 , . . .
, m , we have
[Equation]
For j, j ′ = 1 , . . .
, m , we have
[Equation]
Proof of Proposition 1.
From a probabilistic point of view, assuming that the input variables are random variables, i.e. ( X i , i ∈ J ∪ { i ⋆ } ) are uniformly distributed on [0 , 1] and independent, then (13) can be rewritten as an expectation,
[Equation]
) − (cid:88) i ∈J∪{ i } (cid:98)
Y i ⋆ ,i ( X i ) (cid:19) (cid:33) , where, for i ∈ J , (cid:98)
Y i
=
(cid:80) m i j = (cid:98) ξ i,j ϕ i,j , and for i ∈ J ∪ { i ⋆ } , (cid:98)
Y i ⋆ ,i = (cid:80) m i
j = (cid:101) ξ i,j ϕ i,j (with m i ⋆ = 2 ).
Then
I S ,i ⋆ =
Var (cid:18) (cid:88)
i ∈J (cid:98)
Y i ( X i ) − (cid:88) i ∈J∪{ i } (cid:98)
Y i ⋆ ,i ( X i ) (cid:19) + E(cid:18) (cid:88) i ∈J (cid:98)
Y i ( X i ) − (cid:88) i ∈J∪{ i } (cid:98)
Y i ⋆ ,i ( X i ) (cid:19) =
(cid:88) i ∈J
Var (cid:18) m i (cid:88)
j = ( (cid:98) ξ i,j − (cid:101)
ξ i,j ) ϕ i,j ( X i ) (cid:19)
+ Var (cid:18) (cid:88)
j = (cid:101)
ξ i ⋆ ,j ϕ i ⋆ ,j ( X i ⋆ ) (cid:19) (18) + (cid:18) (cid:88) i ∈J m i (cid:88)
j = ( (cid:98) ξ i,j − (cid:101)
ξ i,j ) E ( ϕ i,j ( X i )) − (cid:88)
j = (cid:101)
ξ i ⋆ ,j E ( ϕ i ⋆ ,j ( X i ⋆ )) (cid:19) .
Recall η i,j = (cid:98) ξ i,j − (cid:101) ξ i,j .
We have for i ∈ J ,
Var (cid:18) m i (cid:88)
j = η i,j ϕ i,j ( X i ) (cid:19) =
E (cid:32) (cid:18) m i (cid:88)
j = η i,j ϕ i,j ( X i ) (cid:19) (cid:33) − (cid:32)
E (cid:18) m i (cid:88)
j = η i,j ϕ i,j ( X i ) (cid:19) (cid:33) =
m i (cid:88) j,j ′ =
η i,j η i,j ′ E ( S i ) j,j ′ − (cid:18) m i (cid:88)
j = η i,j E ( S i ) j (cid:19) , (19) with the notation of Lemma 1.
To compute the term relative to dimension i ⋆ in (18), recall that, if x belongs to the support [0 , 1] of X i ⋆ , then ϕ i ⋆ , 1 ( x ) = 1 − x and ϕ i ⋆ , 2 ( x ) = x .
Hence,
Var (cid:18) (cid:88) j = (cid:101) ξ i ⋆ ,j ϕ i ⋆ ,j ( X i ⋆ ) (cid:19) = Var( (cid:101) ξ i ⋆ , 1 (1 − X i ⋆ )
+ (cid:101)
ξ i ⋆ , 2 X i ⋆ ) = ( (cid:101) ξ i ⋆ , 2 − (cid:101) ξ i ⋆ , 1 ) .
Using (19) and (20) in (18), and observing that E ( ϕ i ⋆ ,j ( X i ⋆ ))
= 1 / 2 for j = 1 , 2 , concludes the proof.
[Table]
Proof of Proposition 2.
As in the proof of Proposition 1, we write
[Equation]
where, for i ∈ J , (cid:98)
[Equation]
(cid:80)
[Equation]
, and (cid:98)
[Equation]
(cid:101) ϕ
[Equation]
,j , where (cid:101) ϕ i,j = ϕ i,j for i ̸ = i ⋆ and where
[Equation] ,j is defined as in ϕ j in Lemma 1 from the subdivision (cid:101)
[Equation]
We express (cid:98)
Y i ⋆ from the current subdivision S i ⋆ to the refined subdivision
[Equation] S i ⋆ , as in Proposition
SM2.1 in [21], which yields
[Equation]
Then we can carry out the same computations as in the proof of Proposition 1 (as if we had (cid:101)
[Equation]
i ⋆ ,j = 0 for j = 1 , 2 in that proof) to conclude, also using Lemma 1.
To enrich the discussion on the robustness of MaxMod in the presence of a non-additive component, we performed a new experiment with a target function given by [Equation] with d = 3 and D = 10 .
Here, λ ≥ 0 is a parameter that controls the influence of the non-additive contribution.
Observe that, as λ increases, the influence of x D − 1 and x D also increases.
We repeat the experiment in Section 5.2.1 for λ = 0 , 0 . 5 , 1 , 1 . 5 , 1 . 7 and for n = 10 D (value also used in Table 2).
The values of λ correspond to percentages of variance explained by the interaction term (Sobol index).
For the Sobol analysis, we have used the R package sensitivity [35].
From
Table 3, we observe that MaxMod properly activates dimensions ( x 1 , x 2 , x 3 , x 9 , x 10 ) in the first iterations while preserving accurate Q 2 values.
For λ > 2 , corresponding to a Sobol index larger than 20% for the interaction term, the Q 2 performance of MaxMod decreases.
This is expected since the additive cGP is not able to capture the non-additive behavior.
A.5 Real application: flood study of the Vienne river (France) A.5.1 Database description
The database consists of numerical simulations using the software Mascaret [34], which is a 1dimensional free surface flow modeling industrial solver based on the Saint-Venant equations [36, 37].
It is composed of an output H representing the water level and 37 inputs depending on a value of flow upstream that is disturbed by a value dQ , on data of the geometry of the bed that are disturbed by modifying the gradients of quantities dZ ref , and on Strickler friction coefficients ( cf 2 for the major bed and cf 1 for the minor bed).
More precisely, the inputs correspond to:
• 12 Strickler coefficients corresponding to cf 1 , denoted as X 1 , . .
, X 12 , whose distributions are uniform over [20 , 40] ; • 12 Strickler coefficients corresponding to cf 2 , denoted as X 13 , . . .
, X 24 , whose distributions are uniform over [10 , 30] ; • 12 dZ ref gradient perturbations, denoted as X 25 , . . .
, X 36 , with standard normal distributions truncated on [ − 3 , 3] ; • and 1 upstream flow disturbance value dQ , denoted as X 37 , with a centered normal distribution with standard deviation σ = 50 and truncated over [ − 150 , 150] .
In [34], the laws (either uniform or truncated Gaussian) of the input parameters have been chosen arbitrarily, according to the empirical distributions observed during experimental campaigns.
Moreover, these random variables have been assumed independent.
Developments in Section 4.2 assume that the input variables are independent and uniformly distributed.
To account for laws different from the uniform one (e.g. the truncated Gaussian law), new analytic expressions of the squared-norm criterion imply more technical developments that are not provided in this work.
However, the expectations in Appendix A.3 can still be approximated via Monte Carlo.
In our numerical experiments, we preferred to apply a quantile transformation of the input space for having independent uniformly distributed random variables X 1 , . . .
, X 37 on [0 , 1] 37 .
It can be shown that this transformation preserves the monotonicity constraints and additive structure.
A.5.2 Additional results
Figures 5 and 6 show the additional results discussed in Section 5.2.2 for n = 3 d and n = 4 d .
[Table] [Table] [Table] [Table]
M a x
M o d
I t e r a t i o n reps [-1 2 3 4 5 6 7 8 
-][Table]
M a x
M o d
I t e r a t i o n
[Figure]
og
[Figure]
Qn =n =n =