Original: Fast variable selection makes scalable Gaussian process BSS-ANOVA a speedy and accurate choice for tabular and time series regression
Revised: Fast variable selection makes Karhunen-Loève decomposed Gaussian process BSS-ANOVA a speedy and accurate choice for dynamic systems identiﬁcation

Original: Abstract
Revised: Abstract

Original: Gaussian processes (GPs) are non-parametric regression engines with a long history.
Revised: Many approaches for scalable GPs have focused on using a subset of data asinducing points.

Original: They are often overlooked in modern machine learning contexts because ofscalability issues: regression for traditional GP kernels are O ( N 3 ) where N is thesize of  the dataset. One of a number ofscalable GP approaches is theKarhunenLoéve (KL) decomposed kernel BSS-ANOVA, developed in 2009.
Revised: Another promising approach is the Karhunen-Loève (KL) decomposition, in which the GP kernel is represented by a set of basis functions whichare the eigenfunctions of the kernel operator. Such kernels have the potential to bevery fast, and do not depend on the selection of a reduced set of inducing points.

Original: It is O ( NP )in training and O ( P ) per point in prediction, where P is the number of terms inthe ANOVA / KL expansion. A new method of forward variable selection, enabledby the ordered nature of the basis functions and fast Gibbs sampling, quickly andeffectively limits the number of terms, yielding a method with competitive accuracies, training and inference times forlarge tabular datasets.
Revised: However KL decompositions lead to high dimensionality, and variable selectionthus becomes paramount. This paper reports a new method of forward variableselection, enabled by the ordered nature of the basis functions in the KL expansionof the Bayesian Smoothing Spline ANOVA kernel (BSS-ANOVA), coupled withfast Gibbs sampling in a fully Bayesian approach. It quickly and effectively limitsthe number of terms, yielding a method with competitive accuracies, training andinference times for tabular datasets of low feature set dimensionality.

Original: The new algorithmdetermines how high the orders of included terms should reach, balancing modelﬁdelity with model complexity using  Bayesian and Akaike information criteria(BIC/AIC).
Revised: The new algorithm determines how high the orders of included terms should reach, balancingmodel ﬁdelity with model complexity using L 0 penalties inherent in Bayesian and Akaike information criteria.

Original: The inference speed and accuracy makes the method especially usefulfor modeling dynamic systems in a model-free manner, by modeling the derivativein a dynamic system as a static problem, then integrating the learned dynamicsusing a high-order scheme.
Revised: The inference speed and accuracy makes the methodespecially useful for modeling dynamic systems, by modeling the derivative in adynamic system as a static problem, then integrating the learned dynamics usinga high-order scheme.

Original: The methods are demonstrated on a ‘Susceptible, Infected, Recovered’ (SIR) toy problem, with the transmissibility used as forcingfunction, along with the  ‘Cascaded Tanks’ benchmark dataset.
Revised: The methods are demonstrated on two dynamic datasets:a ‘Susceptible, Infected, Recovered’ (SIR) toy problem, with the transmissibilityused as forcing function, along with the experimental ‘Cascaded Tanks’ benchmarkdataset.

Original: Comparisons on thestatic prediction of derivatives are made with a Random Forest and Residual Neural
Revised: Comparisons on the static prediction of derivatives are made with a random forest (RF), a residual neural network (ResNet), and the Orthogonal Additive

Original: Network, while for the timeseries prediction comparisons are made with LSTMand GRU recurrent neural networks.
Revised: Kernel (OAK) inducing points scalable GP, while for the timeseries prediction comparisons are made with LSTM and GRU recurrent neural networks (RNNs). The

Original: The GP outperforms the other methods in allmodeling tasks onaccuracy, while (in the case of the neural networks) performingmany orders of magnitude fewer calculations.
Revised: GP outperforms the RF and ResNet on the static estimation, and is comparable to OAK. In dynamic systems modeling it outperforms both RNNs, while performingmany orders of magnitude fewer calculations.

Original: For the SIR test, which involvedprediction for a set of forcing functions qualitatively different from those appearingin the training set, the GP captured the correct dynamics while the neural networksfailed to do so.
Revised: For the SIR test, which involvedprediction for a set of forcing functions qualitatively different from those appearingin the training set, BSS-ANOVA captured the correct dynamics while the neuralnetworks failed to do so.

Original:  BSS-ANOVA Gaussian processes
Revised:  Karhunen-Loève decomposed Gaussian processes

Original: 1.1 Gaussian process fundamentals
Revised: 1.1 Gaussian process fundamentals

Original: Gaussian processes (GPs) are stochastic functions that are engines for nonparametric regression.
Revised: Gaussian processes (GPs) are stochastic functions that are engines for nonparametric regression.

Original: Initially developed for modeling and interpolation in geographic information systems datasets,applications have multiplied across many ﬁelds of data science, including and especially modelaided interpretation of data[Kennedy and O’Hagan, 2001].
Revised: Initially developed for modeling and interpolation in geographic information systems datasets, applications have multiplied across many ﬁelds of data science.

Original: A key advantage of the GP is its broad,continuous nonparametric support and the frequent amenability of different GP kernels to preciseanalysis.
Revised: A key advantage of the GP is itsbroad, continuous nonparametric support and the frequent amenability of different GP kernels toprecise analysis.

Original: A GP is Gaussian in that it is a covariance model linking pairs of points on functional draws.
Revised: A GP is Gaussian in that it is a covariance model linking pairs of points on functional draws.

Original: As sucha GP is completely described by a mean function (often zero in the prior) and covariance kernel.
Revised: As sucha GP is completely described by a mean function (often zero in the prior) and covariance kernel.

Original: Themost famous and perhaps simplest of the covariance kernels is the squared exponential:
Revised: Themost famous and perhaps simplest of the covariance kernels is the squared exponential:

Original: [Equation]
Revised: [Equation]

Original: where the sill ς 2 and range ξ parameters determine the scale and smoothness of the draws.
Revised: where the sill ς 2 and range ξ parameters determine the scale and smoothness of the draws.

Original: In a typicalimplementation modeling a static dataset Z , the statistical model
Revised: In a typicalimplementation modeling a static dataset Z , the statistical model

Original: [Equation]
Revised: [Equation]

Original: with (cid:15) an observation error process, is ﬁrst used to infer the hyperparameters, after which predictionsconditioned on the training dataset can be made.
Revised: with (cid:15) an observation error process, is ﬁrst used to infer the hyperparameters, after which predictionsconditioned on the training dataset can be made.

Original: The draws on the squared exponential GP – alimiting case of the Matérn covariance family – are inﬁnitely differentiable in principle.
Revised: The draws on the squared exponential GP – alimiting case of the Matérn covariance family – are inﬁnitely differentiable.

Original: 1.2 Scalable Gaussian processes and BSS-ANOVA

Original: From a practical standpoint the training of the above GP is O ( N 3 ) , requiring a Cholesky decomposition of the full covariance matrix.
Revised: From a practical standpoint the training of the above GP is O ( N 3 ) , requiring a Cholesky decomposition of the full covariance matrix.

Original: This limits the use of the GP to moderately-sized datasets,generally of a thousand instances or fewer.
Revised: This limits the use of the GP to moderately-sized datasets,generally of a thousand instances or fewer.

Original:  Scalable GPs aim to overcome this difﬁculty through a number of strategies.
Revised: 1.2 Scalable Gaussian processes with inducing points

Original: Liu et al.
Revised: Liu et al.

Original: [2020] providea thorough overview of efforts that aim to improve scalability while maintaining prediction accuracy. Many techniques use global kernel approximations derived in some sense from a set of M << 
Revised: [2020] provide a thorough overview of efforts that aim to improve scalability whilemaintaining prediction accuracy using global kernel approximations derived in some sense from aset of M << N inducing points

Original: Ninducing points [Chalupka et al., 2013, Quinonero-Candela and Rasmussen, 2005,Deisenroth and
Revised: [Chalupka et al., 2013, Quinonero-Candela and Rasmussen, 2005,

Original:  Ng, 2015, Rasmussen and Ghahramani, 2001, Wang et al., 2022] where the goal is to approximate thefull-rank kernel matrix with local approximations.
Revised: Deisenroth and Ng, 2015, Rasmussen and Ghahramani, 2001, Wang et al., 2022]. Generally the goalis to approximate the full-rank kernel matrix with local approximations.

Original: Of particular note is a O ( N ) method that directlyestimates the covariance with training and inference times that limits the increase in M for large Ndeveloped by Wilson et al.
Revised: Of particular note is a O ( N )method that directly estimates the covariance with training and inference times that limits the increasein M for large N developed by Wilson et al.

Revised: Some methods employ ANOVA decompositionsto the full kernel which break out contributions in terms of features and their combinations:

Revised: [Equation]

Revised: 3) which presents opportunities for variable selection [Duvenaud et al., 2011]; of particular note isthe recent Orthogonal Additive Kernel (OAK) which orthogonalizes the kernels in (3) in order tominimize overlap between main effects and higher-order interactions [Lu et al., 2022].

Revised: 1.3 Karhunen-Loève decomposition and BSS-ANOVA

Original: In a distinct line of research from the inducing points-based developments in scalable GPs, Reichet al.
Revised: Another approach to scalability in GPs that is distictive to the inducing points approach is the

Original: [2009] introduced the decomposable Bayesian Smoothing Spline ANOVA (BSS-ANOVA)kernel, whichis subject to ﬁrst an ANOVA decomposition among the inputs (main effects, twoway interactions, etc.) followed by a Karhunen-Loéve (KL) decomposition.
Revised: Karhunen-Loève (KL) expansion, in which the kernel is expressed in terms of a sum over itseigenfunctions: [Equation] ) where [Equation]

Original: This yields a set ofspectral, nonparametric, orthogonal, ordered and – crucially – deterministic basis functions that areprecomputable when inputs are normalized on a [0,1] interval.
Revised: [Equation] Such methods have the potential to be fast: O ( NP ) in training and P per point for inference, where P is the number of terms in the expansion. However such kernels have not been the subject of muchresearch in machine learning contexts generally.

Original: The (non-stationary) kernelfor maineffects is:
Revised: The main issues are tractable calculation of the basisfunctions { φ i } and dimensionality issues [Greengard and O’Neil, 2021]. In 2009 Reich et al. [2009] introduced the Bayesian Smoothing Spline ANOVA (BSS-ANOVA)kernel, which is subject ﬁrst to an ANOVA decomposition, followed by a KL decomposition. Thecore of the BSS-ANOVA kernel is:

Original: [Equation]
Revised: [Equation]

Revised: where B k is the k th Bernoulli polynomial, deﬁned by the generating function

Original: [Equation]
Revised: [Equation]

Original: ) where B k is the k th Bernoulli polynomial.
Revised: 9) yielding [Equation] [Equation]

Original: This kernel is effectively a sum of a non-stationaryquadratic response surface (corresponding to the ﬁrst two terms) and a stationary deviation (the ﬁnalterm).
Revised: This kernel is effectively a sum of a non-stationary quadratic response surface – corresponding tothe ﬁrst two terms in (8) – and a stationary deviation (the ﬁnal term).

Original: Covariances for higher-order interactions are constructed with dyadic products of the maineffect covariance:
Revised: As in (3), covariances forhigher-order interactions are constructed with dyadic products of the main effect covariance:

Original: [Equation]
Revised: [Equation]

Original: such that the full kernel for a system with n inputs is written δ ∼ MV N (0 , Γ) , with
Revised: (13) and so on for higher-order interactions. Terms are then multiplied by scaling hyperparameters andadded together to produce the full kernel:

Original: [Equation]
Revised: [Equation]

Original: The kernel so constructed is supported by a second-order Sobolev space [Reich et al., 2009], which isa very broad and dense class of  functions.
Revised: The kernel so constructed is supported by a second-order Sobolev space [Reich et al., 2009], which isa very broad and dense set of continuous functions.

Revised: Building the kernel in this fashion effectively addresses the problem of generating the eigenfunctionsfrom the KL decomposition: because all of the terms in (14) are based on the generative kernel (8),

Original: The KL decomposition of 5 yields an expansion
Revised: The KL decomposition of 14 will depend only on eigenfunctions of κ 1 .

Original: [Equation] with φ the deterministic basis functions, precomputable if inputs are normalized to  a [0,1] interval.
Revised: Additionally if all inputfeatures are normalized to an [0 , 1] interval (we restrict the discussion to continuous input featuresfor now), then it is only necessary to compute a single set of basis functions { φ i } . The decomposed

Original: Given a statistical model
Revised: BSS-ANOVA GP is written:

Original: [Equation]
Revised: [Equation]

Original: with (cid:15) a white noise observation error, and an appropriate cutoff to the KL expansion 6, the model islinear in the coefﬁcients β .

Original: 1.3 Gibbs sampling

Original: Given the assumption
Revised: Given the assumption

Original: [Equation]
Revised: [Equation]

Original:  then the priors for the coefﬁcients  are independent normal
Revised: 16) then the priors for the coefﬁcients β are iid normal

Original: [Equation]
Revised: [Equation]

Original: where λ k is the eigenvalue for the eigenfunction φ k . For convenience we multiply each basis functionby its eigenvalue such that β iid ∼ N (0 , τ 2 ) . (Basis functions are then evaluated on a dense grid of 500intervals, and ﬁt to cubic splines.)
Revised: Following [Reich et al., 2009] we generate the set { φ i } by producing κ 1 for a dense grid consistingof 500 intervals on [0 , 1] , eigendecompose and ﬁt to cubic splines.

Original: If the variance of the white noise observation error is σ 2 , withinverse gamma prior σ 2 ∼ IG ( a, b ) , with a and b the shape and scale parameters, respectively; and ifτ 2 has inverse gamma prior τ 2 ∼ IG ( a τ , b τ ) , then an iterative Gibbs sampler can be devised suchthat for ﬁxed { σ 2 , τ 2 } , β ∼ MV N ( µ, Σ) , with

Original: [Equation]

Original: [Equation] 11) where X ∈ R N × P is a matrix constructed from the basis functions, whose rows correspond toinstances and columns to terms in the expansion. For ﬁxed { β, τ 2 } , σ 2 ∼ IG ( a ∗ , b ∗ ) , with [Equation]
Revised: Figure 1 shows the ﬁrst 6 basisfunctions.

Original: [Equation]

Original: For ﬁxed { β, σ 2 } , τ 2 ∼ IG ( a ∗ τ , b ∗ τ ) , with [Equation] [Equation] This algorithm is implemented in the routine ‘gibbs_Xin’ in  the supplement.
Revised: These basis functions are nonparametric, pairwise orthogonal, and ordered: note theincrease in frequency and decrease in amplitude as the orders increase.

Original:  Variable selection
Revised:  Variable selection

Original: It’s clear from 6 that the number of terms in the expansion can increase rapidly, even for lowdimensional input spaces.
Revised: It’s clear from (15) that the number of terms in the expansion can increase rapidly, even for lowdimensional input spaces.

Original: A key component of applying the GP to a modeling problem is  theselection of terms.
Revised: A key component of applying the GP to a modeling problem is thus theselection of terms.

Revised: Effectively we seek to minimize the objective function

Revised: [Equation]

Revised: 18) where ζ is a penalty function which leads to a sufﬁciently sparse solution.

Original: 2.1 Indicator variable methods
Revised: 2.1 Indicator variable methods

Original: Reich, Storlie and Bondell took a hierarchical  approach to the problem, estimating a separate varianceτ 2 for each term in the expansion, which is in turn expressed in terms of an indicator variablewith a Bernoulli prior [Reich et al., 2009].
Revised: [Figure] Reich et al. [2009] took a hierarchical Bayesian approach to the problem, estimating a separatevariance τ 2 for each term in the expansion, which is in turn expressed in terms of an indicator variablewith a Bernoulli prior.

Original: This approach, like other ‘indicator variable’ methods,accomplishes the variable selection and the training simultaneously and comprehensively, at the costof requiring a large number of variables in the prior model and a  Markov chain Monte  Carlo (MCMC)sampling procedure.
Revised: This approach, like other ‘indicator variable’ methods, accomplishes thevariable selection and the training simultaneously and comprehensively, at the cost of requiring alarge number of variables in the prior model and a computationally onerous Markov chain Monte [Figure] Carlo (MCMC) sampling procedure.

Original: MCMC is signiﬁcantly slower (and more difﬁcult to implement) than Gibbssampling.
Revised: [Figure] Other sparse optimization methods such as ridge regression or LASSO share the limitation that manyhigh-order terms must be included in the initial model before downselection occurs. [Figure]

Original: 2.2 Forward variable selection
Revised: 2.2 Forward variable selection

Original: Our approach takes advantage of the ordered  nature of the basis functions, using forward variableselection.
Revised: The ordered and orthogonal nature of the basis functions suggests a forward variable selectionapproach. Rewriting the model (15) for a basis function set of maximum order q , [Equation] (19) then considering a model building procedure which increases q stepwise starting with q = 1 revealsthat each subsequent step adds n main effect terms (each depending on a single input), (cid:0) n 2 (cid:1) [2( q − 1)+1]two-way interactions, and (cid:0) n 3 (cid:1) [3( q − 1) 2 + 3( q − 1) + 1] three-way interactions.

Revised: As the modelorder increases the L 2 truncation error for the full kernel decreases as (for the case of a singleinput)[Greengard and O’Neil, 2021]:

Revised: [Equation]

Original:  The algorithm proposes a series of models, using the Bayesian or Akaike Information
Revised: Since the eigenvalues of the BSS-ANOVA kernel decomposition decrease quickly with increasingorder, an approach to the optimization problem (18) focusing on low-order models will sacriﬁce littlein the way of accuracy while realizing signiﬁcant advantages in computing time. The design and implementation of such an approach is the main contribution of this work. Itapproaches the optimization of (18) with an iterative process, ﬁnding the most efﬁcient truncation ofthe system while evaluating the cost function only for candidate models with fewer terms than theoptimum truncation. The method is fully Bayesian, with a fast Gibbs sampling procedure at its core.

Revised: As such the form of the cost function is also Bayesian in nature, taking the form of the Bayesian or

Original: Criteria (BIC/AIC) to determine when to stop the iterations.
Revised: Akaike information criteria (BIC/AIC), which incorporate L 0 penalties. 2.2.1 Gibbs sampling Given a statistical model [Equation]

Revised: (21) with (cid:15) a white noise observation error, and a given truncation to the KL expansion (15), the model islinear in the coefﬁcients β and Gibbs sampling can be used to estimate parameters in a fully Bayesianmethodology.

Revised: If the variance of the observation error is σ 2 , with inverse gamma prior σ 2 ∼ IG ( a, b ) , with a and bthe shape and scale parameters, respectively; and if τ 2 has inverse gamma prior τ 2 ∼ IG ( a τ , b τ ) , then an iterative Gibbs sampler can be devised such that for ﬁxed { σ 2 , τ 2 } , β ∼ MV N ( µ, Σ) , with

Original: The algorithm constructs models with terms having up to (optionally) three-way interactions (that is,terms depending on three different inputs at once).
Revised: [Equation] [Equation] where X ∈ R N × P is a matrix constructed from the basis functions, whose rows correspond toinstances and columns to terms in the expansion. For ﬁxed { β, τ 2 } , σ 2 ∼ IG ( a ∗ , b ∗ ) , with

Revised: [Equation]

Revised: [Equation]

Original: Terms are added in stages labeled by an integer
Revised: [Equation] For ﬁxed { β, σ 2 } , τ 2 ∼ IG ( a ∗ τ , b ∗ τ ) , with [Equation] [Equation]

Revised: [Equation]

Original:  “index” that initializes at 1.
Revised: This algorithm is implemented in the routine ‘gibbs_Xin’ in the supplement. 2.2.2 Optimization algorithm The algorithm constructs models with terms having up to three-way interactions. Terms are addedin stages labeled by an integer “index” that initializes at 1.

Original: At each stage, a series of substages cycle through all permutations of basisfunction orders that sum up to that stage’s index.
Revised: At each stage, a series of substagescycle through all permutations of basis function orders that sum up to that stage’s index.

Original: Stage 1 adds only ﬁrst order main effects.
Revised: Stage 1adds only ﬁrst order main effects.

Original: Stage 2adds second order main effects and ﬁrst order two way interactions – corresponding to φ 1 ( ϑ i ) φ 1 ( ϑ j )– in two separate substages.
Revised: Stage 2 adds second order main effects and ﬁrst order two wayinteractions – corresponding to φ 1 ( x i ) φ 1 ( x j ) – in two separate substages.

Original: The substages always occur such that terms involving lower-order basisfunctions (in the case of stage 2, this is the two-way interactions) come ﬁrst.
Revised: The substages alwaysoccur such that terms involving lower-order basis functions (for example in the case of stage 2, this isthe ﬁrst order two-way interactions) come ﬁrst.

Original: Each substage adds atonce all combinations of inputs and all permutations among each combination.
Revised: Each substage adds at once all combinations of inputsand all permutations among each combination, such that each substage adds (cid:0) n 2 (cid:1) terms for two-wayinteractions and (cid:0) n 3 (cid:1) terms for three-way interactions.

Original: Then the sampler iscalled and the BIC or AIC is calculated.
Revised: Then the sampler is called and the BIC or AICis calculated.

Original: Because there is not a monotonic decrease / increase patternfor the discriminator, a “tolerance” setting controls how many substages the algorithm can iteratethrough without ﬁnding a new minimum BIC or AIC before it terminates.
Revised: Because there is not a monotonic decrease / increase pattern for the objective function,a “tolerance” setting controls how many substages the algorithm can iterate through without ﬁnding anew minimum BIC or AIC before it terminates.

Original: The algorithm returns theoptimum model.
Revised: The algorithm returns the optimum model.

Original: Algorithm 1 BSS
Revised: This algorithm appears in the routine ‘emulator_Xin’ in the supplement.  Experiments: Dynamic system identiﬁcation 3.1 Procedure BSS-ANOVA regression – as is the case for other GPs – is most effective for tabular datasets withcontinuous inputs and targets of moderate dimensionality.

Original: 1: procedure F WD V AR S ELECT ( ϑ , Z , φ , tol, h ) (cid:46) h is a vector of hyperparameters.
Revised: This suggests an application in dynamicsystems identiﬁcation. Indeed BSS-ANOVA GPs have been utilized as components of other models(“intrusively”) for this purpose in a number of applications

Original: 2: ind =3: count =4: while count < tol do 5: if ind is new then 6:
Revised: [Bhat et al., 2017, Lei et al., 2019, Ostaceet al., 2020]. We demonstrate here that they may also be used directly to identify dynamics in moregeneral cases, without the aid of an accompanying model. The procedure is a concurrent one, in that derivatives estimated from the datasets are modeled directlyusing BSS-ANOVA with forward variable selection, using the concurrent values of the system statesand other inputs; for example a two-state system is modeled using two separate GPs: [Equation]

Original: Find all combinations of integers that
Revised: [Equation] The identiﬁed system is then integrated to yield predictions with uncertainty.

Original: sum
Revised: The procedure was demonstrated on two nonlinear dynamic datasets: a synthetic dataset derived fromthe susceptible, infected, recovered model (SIR model) for infectious disease, and the ‘Cascaded 1: procedure F WD V AR S ELECT ( x , Z , φ , tol, h ) (cid:46) h is a vector of hyperparameters. 2: ind =3: count =4: while count < tol do 5: if ind is new then 6:

Original:  up to ind, ordering them by the maximum 7: integer appearing in each combination, with the lowest maximum ﬁrst.
Revised: Find all combinations of integers that sum up to ind, ordering them by the maximum 7: integer appearing in each combination, with the lowest maximum ﬁrst.

Original: Select the next combination in the set and place the integers into a vector with as many  elements as there are model inputs, buffering out with zeroes.
Revised: Select the next combination in the set and place the integers into a vector with as many 9: elements as there are model inputs, buffering out with zeroes.

Original: Produce a matrix M d the rows of which contain all permutations of that vector.
Revised: Produce a matrix M d the rows of which contain all permutations of that vector.

Original: 11: (cid:46)
Revised: 11: (cid:46)

Original: Each row corresponds to a term in the GP expansion.
Revised: Each row corresponds to a term in the GP expansion.

Original: Produce an input matrix X d where columns are model terms and rows are experiments,  for all terms appearing in M d .
Revised: Produce an input matrix X d where columns are model terms and rows are experiments, 13: for all terms appearing in M d .

Original: Recursively concatenate:
Revised: Recursively concatenate:

Original: X =
Revised: X =

Original: [ XX d ] , M =
Revised: [ XX d ] , M =

Original: [ M ; M d ] 
Revised: [ M ; M d ] 15: 5.

Original: β , BIC = gibbs_Xin( X , Z , φ , h ) 16: if the BIC is a minimum for all models then 17: save the model 18: count =19: else 20: count++ 21: if all combinations for ind have been utilized then 22: ind++ 23:
Revised: β , BIC = gibbs_Xin( X , Z , φ , h ) 16: if the BIC is a minimum for all models then 17: save the model 18: count =19: else 20: count++ 21: if all combinations for ind have been utilized then 22: ind++ 23:

Original: Return M , β , BIC
Revised: Return M , β , BIC

Original: This algorithm appears in the routine ‘emulator_Xin’ in the supplement.

Original: Experiments and results: dynamic system identiﬁcation

Original: 3.1 Procedure

Original: BSS-ANOVA regression – as is the case for other GPs – is most effective for tabular datasets withcontinuous inputs and targets of moderate dimensionality.

Original: This suggests an application in dynamicsystem identiﬁcation.

Original: Indeed BSS-ANOVA GPs have been utilized as components of other models(“intrusively”) for this purpose in a number of applications

Original: [Bhat et al., 2017, Lei et al., 2019, Ostaceet al., 2022].

Original: We demonstrate here that they may also be used directly to identify dynamics in moregeneral cases, without the aid of an accompanying model.

Original: The procedure is a concurrent one, in that derivatives estimated from the datasets are modeled directlyusing BSS-ANOVA with forward variable selection, using the concurrent values of the system statesand other inputs; for example a two-state system is modeled using two separate GPs:

Original: [Equation]

Original: [Equation]

Original: The identiﬁed system is then integrated to yield predictions with uncertainty.

Original: The procedure was demonstrated on two nonlinear dynamic datasets: a synthetic dataset derived fromthe susceptible, infected, recovered model (SIR model) for infectious disease, and the ‘Cascaded

Original: [Table]

Original: [Table]

Original: Tanks’ experimental benchmark dataset.
Revised: Tanks’ experimental benchmark dataset.

Original: In both cases comparisons were made to long short termmemory (LSTM) and gated recurrent unit (GRU) neural netowrks for timeseries prediction.
Revised: In both cases comparisons were made to long short termmemory (LSTM) and gated recurrent unit (GRU) neural netowrks for timeseries prediction.

Original: In thecase of the cascaded tanks benchmark comparisons were made against random forest (RF) and aresidual neural network (RNN) for the static derivative estimation problem.
Revised: In thecase of the cascaded tanks benchmark comparisons were made against random forest (RF), a residualneural network (ResNet) and the state-of-the-art OAK inducing points scalable GP [Lu et al., 2022]for the static derivative estimation problem.

Original: 3.2 Benchmark dataset: cascaded tanks
Revised: 3.2 Experimental benchmark: Cascaded tanks

Original: The cascaded tanks nonlinear benchmark dataset is an experimental nonlinear dynamic system[Wigren and Schoukens, 2013].
Revised: The cascaded tanks nonlinear benchmark dataset is an experimental nonlinear dynamic system[Wigren and Schoukens, 2013].

Original: The experiment consists of a set of two tanks and a reservoir of water.
Revised: The experiment consists of a set of two tanks and a reservoir of water.

Original: An upper tank is ﬁlled by a pump from the reservoir.
Revised: An upper tank is ﬁlled by a pump from the reservoir.

Original: An outlet in the upper tank empties into thelower tank, which in turn empties through an outlet back into the reservoir.
Revised: An outlet in the upper tank empties into thelower tank, which in turn empties through an outlet back into the reservoir.

Original: A signal sent to the pumpserves as the forcing function for the system, with the tank water level heights the two states of thesystem.
Revised: A signal sent to the pumpserves as the forcing function for the system, with the tank water level heights the two states of thesystem.

Original: A newer version of the dataset includes an ‘overﬂow’ phenomenon which is considered tobe a ‘hard nonlinearity’

Original: [Schoukens and Noel, 2017], however the newer dataset provides only atimeseries for the upper tank h 1 along with the pump signal u .

Original: We chose to demonstrate the novel

Original: GP approach and compare to neural network-based methods on the multivariate dataset.

Original: We ﬁrst compared the performance of the GP with RF and RNN static regressors.
Revised: We ﬁrst compared the performance of BSS-ANOVA with RF, ResNet and OAK static regressors.

Original: Derivatives werecalculated via direct ﬁnite differences for the relatively noise-free dataset, yielding 10000 instances.
Revised: Derivatives were calculated via direct ﬁnite differences for the relatively noise-free dataset, yielding10000 instances.

Original: Each method was trained on concurrent values of both states and the forcing function for eachderivative.
Revised: Each method was trained on concurrent values of both states and the forcing functionfor each derivative.

Original: For the GP we used hyperparameters of a = 1000 , b = 1 . 001 , a τ = 4 and b τ = 55 for˙ h 1 and 69.1 for ˙{+69.
+}
Revised: For the GP we used hyperparameters of a = 1000 , b = 1 . 001 , a τ = 4 andb τ = 55 for ˙ h 1 and {+69.
+}

Original:  h 2 , with tolerances of 3 for ˙ h 1 and 5 for ˙ h 2 , and the AIC as discriminator.
Revised: for ˙ h 2 , with tolerances of 3 for ˙ h 1 and 5 for ˙ h 2 , and the AIC as discriminator.

Original: Of 2000draws the ﬁrst 1000 were discarded.
Revised: Of 2000 draws the ﬁrst 1000 were discarded.

Original: Only two-way interactions were required.
Revised: Only two-way interactions were required.

Original: For the RF 100 treeswere used with a leaf size of 5.
Revised: For the RF100 trees were used with a leaf size of 5.

Original: The RNN had a depth of 6 (ﬁlter sizes ranging from 16 to 64) and inbetween each fully connected layer is a batch normalization and relu layer.
Revised: The ResNet had a depth of 6 (ﬁlter sizes ranging from 16to 64) and in between each fully connected layer is a batch normalization and relu layer.

Original: The mini batch size is 16,initial learn rate is 0.001, the data was shufﬂed every epoch for a total of 30 epochs, and the validationfrequency was 1000.
Revised: The minibatch size is 16, initial learn rate is 0.001, the data was shufﬂed every epoch for a total of 30 epochs,and the validation frequency was 1000.

Revised: OAK was applied at a maximum dimension of 3 and withthe default value of 200 inducing points.

Original: The 5-fold cross-validated results appear in Table 1.
Revised: The 5-fold cross-validated results appear in Table 1.

Original: The GP performed best forboth outputs by clear margins, followed  by  the RF and then the RNN.
Revised: OAKperformed best for both outputs, followed closely by BSS-ANOVA. Both GPs outperformed the RFand the ResNet by clear margins.

Original: Timeseries predictions follow for the GP via a 4 th -order Runge-Kutta integration routine.
Revised: Timeseries predictions follow for the GP via a 4 th -order Runge-Kutta integration routine.

Original: These werecompared with LSTM and GRU recurrent neural networks.
Revised: These werecompared with LSTM and GRU recurrent neural networks (RNNs).

Original: For the LSTM there was one LSTM layerand a total of 128 hidden layers, the data was shufﬂed every epoch for a maximum of 125 epochs,verbose was equal to 0, and the sequence was padded to the left.
Revised: For the LSTM there was one LSTM layer and a total of 128 hidden layers, the data was shufﬂed every epoch for a maximum of125 epochs [Table]

Original: The GRU had one GRU layer and150 total hidden layers, the data was shufﬂed every epoch for a total of 150 epochs, verbose wasequal to zero and the sequence was padded to the left.
Revised: [Table] GRU layer and 150 total hidden layers, the data was shufﬂed every epoch for a total of 150 epochs,verbose was equal to zero and the sequence was padded to the left.

Original: The 5-fold cross-validated results (datapointswere not randomized before creating the folds so as to preserve the timeseries order) appear in Table2.
Revised: The 5-fold cross-validated results(datapoints were not randomized before creating the folds so as to preserve the timeseries order)appear in Table 2.

Original: The GP is most accurate, followed by the LSTM and the GRU.
Revised: BSS-ANOVA is most accurate, followed by the LSTM and the GRU.

Original: Figure 1 shows the predictions ofthe GP and the LSTM for the upper tank for one of the test folds.
Revised: Figure 2shows the predictions of the GP and the LSTM for the upper tank for one of the test folds.

Original: The GP predictions are superiornear the sharp inﬂection and critical points where nonlinearities are strongest.
Revised: The GPpredictions are superior near the sharp inﬂection and critical points where nonlinearities are strongest.

Original: Note that the ﬁrst 50points of each test set, which were provided to the LSTM and GRU as a start-up set in the predictionphase, were removed from the calculation of error for both methods.
Revised: Note that the ﬁrst 50 points of each test set, which were provided to the LSTM and GRU as a start-upset in the prediction phase, were removed from the calculation of error for both methods.

Revised: While it is reasonable to expect that OAK with 200 inducing points would outperform BSS-ANOVAin the time integration, it was not practical to make this comparison for reasons of computing time.

Revised: Acomparison with a reduced number of inducing points and increased time step in the integrator wasmade – results are discussed in section 3.4.

Original: 3.3 Synthetic dataset: susceptible, infected, recovered model
Revised: 3.3 Synthetic benchmark: Susceptible, infected, recovered model

Original: The susceptible, infected, recovered model (SIR model) is a common simulation for infectious disease.
Revised: The susceptible, infected, recovered model (SIR model) is a common simulation for infectious disease.

Original: Though there are several versions, the simplest is three states, only two of which are independent.
Revised: Though there are several versions, the simplest is three states, only two of which are independent.

Original: The system is written
Revised: The system is written

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: 21) where S ( t ) is the susceptible population, I ( t ) the infected, R ( t ) the recovered, B ( t ) is the transmissibility rate (which we utilize as a forcing function), γ is the recovery rate (which we leave ﬁxed at0.5) and N P is the total population.
Revised: where S ( t ) is the susceptible population, I ( t ) the infected, R ( t ) the recovered, B ( t ) is the transmissibility rate (which we utilize as a forcing function), γ is the recovery rate (which we leave ﬁxed at0.5) and N P is the total population.

Original: Because N P is ﬁxed, only two states are independent, so thesystem dynamics can be captured by modeling only two of the three.
Revised: Because N P is ﬁxed and S + I + R = N P , only two states areindependent, so the system dynamics can be captured by modeling only two of the three.

Original: We chose I ( t ) and R ( t ) .
Revised: We chose I ( t ) and R ( t ) .

Original: The training data consists of 58 curves.
Revised: The training data consists of 58 curves.

Original: All curves in the training set have a ﬁxed B value rangingfrom 0.5 to 9, in six intervals of 1.7.
Revised: All curves in the training set have a ﬁxed B value rangingfrom 0.5 to 9, in six intervals of 1.7.

Original: For each value of B there are 8-10 siumulations correspondingto different initial conditions designed in such a way to provide coverage of the state space.
Revised: For each value of B there are 8-10 siumulations correspondingto different initial conditions designed in such a way to provide coverage of the state space.

Original: (Exactinitial conditions used appear in the supplement.)
Revised: (Exactinitial conditions used appear in the supplement.)

Original: Each simulation used N P = 1000 .
Revised: Each simulation used N P = 1000 .

Original: The test data consists of 24 curves, each of which features a temporally changing transmissibility
Revised: The test data consists of 24 curves, each of which features a temporally changing transmissibility

Original: B ( t ) .
Revised: B ( t ) .

Original: There are three initial B 0 values: 1.35, 4.75 and 8.15.
Revised: There are three initial B 0 values: 1.35, 4.75 and 8.15.

Original: For each starting point there are twotypes of transmissibility curves: a ramp and a sinusoid.
Revised: For each starting point there are twotypes of transmissibility curves: a ramp and a sinusoid.

Original: The B 0 = 1 . 35 and B 0 = 4 .
Revised: The B 0 = 1 . 35 and B 0 = 4 .

Original: 75 startingpoints have ramps with a positive slope of 1, while the B 0 = 8 .
Revised: 75 startingpoints have ramps with a positive slope of 1, while the B 0 = 8 .

Original: 15 curves have a slope of -1.
Revised: 15 curves have a slope of -1.

Original: Allramps run from t=0 to t=4, where they level off.
Revised: Allramps run from t=0 to t=4, where they level off.

Original: The sinusoids have amplitudes between 0.5 and 3and a period of 1.
Revised: The sinusoids have amplitudes between 0.5 and 3and a period of 1.

Original: Hyperparameters for the GP were: a = a τ = 4 for both states, b τ,R = 8 .
Revised: Hyperparameters for BSS-ANOVA were: a = a τ = 4 for both states, b τ,R = 8 .

Original: 95 and b τ,I = 72 .
Revised: 95 and b τ,I = 72 .

Original: 1 , whileb I = 1 .
Revised: 1 ,while b I = 1 .

Original: 25 and b R = 20 . 2000 draws were taken and the ﬁrst 1000 discarded.
Revised: 25 and b R = 20 . 2000 draws were taken and the ﬁrst 1000 discarded.

Original: The tolerance was 6.
Revised: The tolerancewas 6.

Original: Hyperparameters for the LSTM and GRU were the same as for the Cascaded Tanks.
Revised: Hyperparameters for the LSTM and GRU were the same as for the Cascaded Tanks.

Original: A partial display of the results are shown in Figures 2 for the GP and 3 for the GRU, which wasthe better performing of the two neural nets on this dataset.
Revised: A partial display of the results are shown in Figures 3 for BSS-ANOVA and 4 for the GRU, whichwas the better performing of the two neural nets on this dataset.

Original: For the GP, the total test set MAE was5.2739 ± 4.0138 for I and 11.8345 ± 21.7337 for R , corresponding to MAPEs of 8.99 ± 4.92 for I and2.80 ± 2.52 for R .
Revised: For the GP, the total test set MAEwas 5.2739 ± 4.0138 for I and 11.8345 ± 21.7337 for R , corresponding to MAPEs of 8.99 ± 4.92 for Iand 2.80 ± 2.52 for R .

Original: Statistics were not calculated for the GRU as it failed to replicate the dynamics inmost test cases and was obviously inferior in a quantitative sense in every instance, as shown in 3.
Revised: Statistics were not calculated for the GRU as it failed to replicate the dynamicsin most test cases and was obviously inferior in a quantitative sense in every instance, as shown in Figure 4. [Figure]

Original: 3.4 Training and inference times
Revised: 3.4 Training and inference times

Original: Training and inference times for the GP were fast, with a mean total train time of 6.3 seconds for thecascaded tanks and 10.8 seconds for the SIR, with 8,000 and 20,000 training data points, respectively,on a 2019 6-core i7 processor with 16 GB of RAM.
Revised: Training and inference times for BSS-ANOVA were fast, with a mean total train time of 6.3 secondsfor the cascaded tanks and 10.8 seconds for the SIR, with 8,000 and 20,000 training data points,respectively, on a 2019 6-core i7 processor with 16 GB of RAM.

Original: The routines were implemented in MATLAB, butnot parallelized or optimized for speed.
Revised: The routines were implemented in MATLAB, but not parallelized or optimized for speed.

Original: Models for ˙ h 1 contain between 23 and 41{+
 [Figure]+} terms, while ˙ h 2has between 38 and 57 terms.
Revised: Models for ˙ h 1 contain between 23 and {+
 [Figure]+} terms, while ˙ h 2 has between 38 and 57 terms.

Original: Prediction times for 2000 static points for the cascaded tanks averages0.5437 s, and the time for evaluating integrals over the test set averages 20.22 s. For the SIR modelthe ˙ I model had 81 terms and the ˙ R model 9 terms, with a mean integration time of 5.3 s.
Revised: Prediction times for 2000 static points for the cascadedtanks averages 0.5437 s, and the time for evaluating integrals over the test set averages 20.22 s. Forthe SIR model the ˙ I model had 81 terms and the ˙ R model 9 terms, with a mean integration time of 5.3s.

Revised: Analyses have shown that the rate limiting step in BSS-ANOVA build algorithms are the O ( NP )construction of the X matrix from the inputs and basis functions.

Original: Neural networks were native MATLAB functions, parallelized and optimized for speed.
Revised: The neural networks were native MATLAB functions, parallelized and optimized for speed.

Original: Nonethelessaverage train times were considerably longer, with mean train times of 130 s for the RNN and175and 123 s, respectively, for training the LSTM and GRU for the cascaded tanks.
Revised: Nonetheless train times were considerablylonger, with mean train times of 130s for the ResNet and 175 and 123 s, respectively, for training the LSTM and GRU for the cascaded tanks.

Original: Analyses have shown that the rate limiting step in the GP build algorithms are  the O ( NP ) constructionof the X matrix from the inputs and basis functions.
Revised: This is to be expected given that the number of weights inthe neural nets are on the order of 10 4 . It was not feasible to integrate OAK at the level of 200 inducing points to the same standard asthat of BSS-ANOVA because of time considerations.

Original: Discussion
Revised: A reduced set of 40 inducing points yieldedaccuracies in the static estimation problem that were approximately the same as BSS-ANOVA. Areduced time step (500 vs. 20,000 integration steps) brought the integration time down to 51 minutesfor OAK, with MAE/MAPE of 0.1554/6.3 for h 1 and 0.2378/9.1 for h 2 . Reducing the integrationstep to the same level as BSS-ANOVA (where we could expect comparable integration accuracies)would require approximately 33 hours.  Limitations and future work

Original: The two examples presented in this paper were the only two attempted for dynamic systems identiﬁcation.
Revised: The two examples presented in this paper were the only two attempted for dynamic systems identiﬁcation.

Original: Other static prediction problems – especially those involving high dimensionality in theinput space with some categorical inputs – are more challenging (the routines perform reasonablywell for the ‘Adult Income’ benchmark for instance but can be exceeded in accuracy by some deep
Revised: Other benchmark dynamic systems, especially those chaotic in nature, will be examinedin future work.

Original: [Figure] learning frameworks and gradient boosted random forests) and require more sophisticated methodsfor variable selection. Future work will address prediction in these large tabular datasets.
Revised: Desipite the advance in variable selection represented by this routine, datasets withhigher dimensionalities in the feature space are more challenging and require additional methodsfor variable selection, which are already in development.

Original: Many moreexperiments and comparisons need to be performed for dynamic systems as well, with larger datasetsand more difﬁcult identiﬁcation problems.
Revised: More experiments and comparisons willbe performed for dynamic systems as well, with larger datasets and more difﬁcult identiﬁcationproblems.

Original: The GP  is inaccurate in extrapolation: when test setinputs exceed the bounds of the training set the resulting extrapolation sometimes causes instabilitiescausing the integration procedure to fail.
Revised: Like any GP BSS-ANOVA is inaccurate in extrapolation: when test set inputs exceedthe bounds of the training set the resulting extrapolation sometimes causes instabilities causing theintegration procedure to fail.

Original: These failures were eliminated by preventing extrapolationeven in instances where the inputs exceeded the bounds, but more stable extrapolation strategies willpossibly become necessary for longer prediction windows where extrapolation is unavoidable.
Revised: These failures were eliminated by preventing extrapolation even ininstances where the inputs exceeded the bounds, but more stable extrapolation strategies will possiblybecome necessary for longer prediction windows where extrapolation is unavoidable.

Original: Given the performance and speed of the GP for dynamic system identiﬁcation, control applicationsare of particular interest.

Original: Fast methods for online updating are possible and should make the methoda strong contender for these applications.

Original: Optimization and parallelization of theroutines could signiﬁcantly improve training and inferencetimes.

Original: Python-based open-source versions of the optimized codes will be made publicly available.

Original: Conclusion

Original: A new forward variable selection algorithm has made the scalable Gaussian process BSS-ANOVA afast and accurate method for nonparametric regression of tabular data on continuous input spaces.

Original: The speed and accuracy for this type of dataset makes it ideal for dynamic system identiﬁcation.

Original: Favorable comparisons with popular neural network approaches for timeseries problems were madeon the basis of both speed and accuracy for nonlinear synthetic and benchmark experimental dynamicdatasets.

Original: [Figure]

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
Revised: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?

Original: [Yes] (b) Did you describe the limitations of your work?
Revised: [Yes](b) Did you describe the limitations of your work?

Original: [Yes] see Section 4(c)
Revised: [Yes] see Section 4(c)

Original: Did you discuss any potential negative societal impacts of your work?
Revised: Did you discuss any potential negative societal impacts of your work?

Original: [N/A]
Revised: [N/A]

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]
Revised: [N/A]3.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material or as a URL)?
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] See the supplement
Revised: [Yes] See thesupplement(b)

Original: (b) Did you specify all the training details (e
Revised: Did you specify all the training details (e

Original: g., data splits, hyperparameters, how they were chosen)?
Revised: g., data splits, hyperparameters, how theywere chosen)?

Original: [Yes] See Sections 3.2 and 3.
Revised: [Yes] See Sections 3.2 and 3.

Original: (c) Did you report error bars (e g., with respect to the random seed after running experi- ments multiple times)?
Revised: (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] See Sections 3.2 and 3.
Revised: [Yes] See Sections 3.2 and 3.

Original: (d) Did you include the total amount of compute and the type of resources used (e
Revised: (d) Did you include the total amount of compute and the type of resources used (e

Original: g., typeof GPUs, internal cluster, or cloud provider)?
Revised: g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Section 3.
Revised: [Yes] See Section 3.

Original: If you are using existing assets (e
Revised: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...
Revised: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] See Section 3.
Revised: [Yes] See Section 3.2(b)

Original: (b) Did you mention the license of the assets?
Revised: Did you mention the license of the assets?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes] See the supplement
Revised: [Yes]See the supplement

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identiﬁableinformation or offensive content?
Revised: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects..

Original: (a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
Revised: (a) Did you include the full text of instructions given to participants and screenshots if

Revised: applicable?

Original: [N/A]
Revised: [N/A

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A](c)
Revised: [N/A]

Original:  Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]