Sparse Gaussian Process Hyperparameters: Optimize or Integrate?
Anonymous Author(s)
AffiliationAddress email
Abstract
The kernel function and its hyperparameters are the central model selection choicein a Gaussian process [Rasmussen and Williams, 2006].
Typically, the hyperparameters of the kernel are chosen by maximising the marginal likelihood, anapproach known as Type-II maximum likelihood (ML-II).
However, ML-II doesnot account for hyperparameter uncertainty, and it is well-known that this canlead to severely biased estimates and an underestimation of predictive uncertainty.
While there are several works which employ a fully Bayesian characterisationof GPs, relatively few propose such approaches for the sparse GPs paradigm.
Inthis work we propose an algorithm for sparse Gaussian process regression whichleverages MCMC to sample from the hyperparameter posterior within the variational inducing point framework of [Titsias, 2009].
This work is closely relatedto Hensman et al.
[2015b], but side-steps the need to sample the inducing points,thereby significantly improving sampling efficiency in the Gaussian likelihood case.
We compare this scheme against natural baselines in literature along with stochasticvariational GPs (SVGPs) along with an extensive computational analysis.
Introduction
Gaussian processes (GPs) are a prominent class of models for supervised learning which can quantifyuncertainty and incorporate inductive biases in function space via the kernel function.
Hand-crafting akernel function is a powerful way to incorporate prior knowledge.
In many instances not all propertiesof a kernel function can be specified from prior knowledge alone and parameters are chosen via MLII.
However, defining a complex kernel function with a large number of hyperparameters can makethe marginal likelihood prone to multiple local optima and overfitting.
Further, several local optimamay correspond to priors that do not sensibly model the data.
Weakly identified hyperparameterscan manifest in flat ridges in the marginal likelihood surface 1 making gradient based optimisationextremely sensitive to starting values [Warnes and Ripley, 1987].
Overall, the ML-II point estimatesfor the hyperparameters are subject to high variability and underestimate prediction uncertainty.
The problem of ridges in the marginal likelihood surface also does not necessarily go away as moreobservations are collected.
For example, if f 1 and f 2 are Brownian motions, σf 1 ( x/ℓ ) is equal indistribution to √ ασf 2 ( x/αℓ ) , which means observations do not provide any information about theproduct σℓ .
More generally, for a greater class of kernels, including the Mat´ern– 1 / 2 kernel, σf 1 ( x/ℓ )is equivalent to √ ασf 2 ( x/αℓ ) , which means that is not possible to consistently estimate the productσℓ from data, even if infinitely many observations are collected in a fixed domain [Chapter 6, Stein,1999].
This implies that one cannot estimate the individual hyperparameters ( σ, ℓ ) consistently.
Italso motivates why there can be benefits to estimating the hyperparameter posterior even in largedata regimes, and ML-II may be insufficient.
A more satisfactory treatment of hyperparametersinvolves placing a prior over the hyperparameters and performing Bayesian inference to compute a values
(hyper)posterior.
For large datasets, this motivates using scalable GP inference (e.g. sparse methods)in conjunction with Markov chain Monte Carlo (MCMC) for the hyperparametrs.
The Bayesian treatment of weakly identified hyperparameters may also be fraught with difficulties.
Gradient-based samplers like Hamiltonian Monte Carlo [Neal et al., 2011] and its variants havedifficulty navigating regions of high curvature and flat ridges where the gradient offers no informationfor transition [Betancourt, 2017].
This leads to over-concentration of samples from the flat region.(Usually, this can at least partially be rectified with informative priors.)
These pathologies are alsotypical of other hierarchical models [Betancourt and Girolami, 2015].
Figure 1 shows that the GPmarginal likelihood surface can manifest these pathologies.
The evidence lower bound (ELBO) usedin hyperparameter selection for variational sparse GPs relying on inducing points inherits similar,or even less favourable, characteristics to the exact marginal likelihood.
As a result, for weaklyinformative priors, gradient based samplers are susceptible to getting stuck at the boundary of thesepathological regions hence biasing the sample estimates.
Fortunately, the effective sample size metricused for diagnoising mixing in MCMC is indicative of this behaviour when directly observing thephase space of the target distribution is infeasible in high-dimensions.
Historical justification for ML-II (also called the evidence framework) comes from [MacKay, 1994]which highlighted several conditions for ML-II to yield reasonable estimates for the hyperparameters.
Crucially, the evidence is unlikely to manifest multiple local optima for a model well-matched to thedata and with a high signal-to-noise ratio.
Transferring this insight to the Gaussian process paradigmwe show how the evidence can have a significant tail or no well-defined maximum in settings with alow signal-to-noise ratio which arises with high aleatoric uncertainty or sparse data, frequently both.
In these settings, a point estimate may not adequately summarise the hyperparameter posterior and thebenefits of marginalisation stand out.
The situation is only exacerbated in high-dimensions (i.e. whenthere are many hyperparameter) where increasingly more volume of the posterior is captured in athin-shell making the density peak extremely unrepresentative of the posterior.
Unlike the likelihood of parameters, the marginal likelihood inherently contains a trade-off betweenthe data-fit and complexity penalty term.
This is one of the main properties that makes the marginallikelihood objective a viable choice for model selection [Rasmussen and Williams, 2006].
Forexample, for the Gaussian process regression model,
[Equation]
the marginal likelihood takes the form,
[Equation]
where c is a constant, p ( y | f ) denotes the data likelihood and θ denotes kernel hyperparameters.
Thistrade-off is a well-established idea which embodies the automatic Occam’s razor effect [Rasmussenand Ghahramani, 2001] where models well-suited to the data are automatically selected just by usingthe marginal likelihood objective.
This may seem to contradict earlier claims regarding overfitting, but by shying away from dealingwith the hyperparameter posterior we risk overfitting even with the marginal likelihood approach.
In other words, the evidence framework subdues the overfitting effect induced by the canonicalmaximum likelihood approach which does not have any complexity penalty term.
The parameters inthis approach are free to fit the data as well as possible making them prone to overfitting and poorgeneralisation.
Overparameterized kernels based on neural networks like deep kernel models [Wilsonet al., 2016] are well known to exacerbate overfitting [Ober et al., 2021].
The main motivation for this work is to highlight that fully Bayesian schemes in sparse Gaussianprocess models are practically beneficial.
While several works in the literature employ fully Bayesianscheme of integrating out the hyperparameters (see table 1), in this work we attempt to analysethem in an orthogonal direction, focusing on comparison with the evidence framework and otherbenchmarks by extending the main sparse variational formulation in the literature Titsias [2009].
We present a generalised inference scheme for fully Bayesian GP regression and counteract someof the computational cost of sampling both inducing variables and hyperparameters by deriving adoubly collapsed bound which selects the optimal distribution over the inducing points analyticallyand targets the kernel hyperparameters with HMC.
Related Work
[Table]
[Table]
and Rasmussen [2020] ✗
[Table]
Rossi
[Table]
[Table]
[Table]
This work
[Table]
HMC /
[Table]
Fully Bayesian Gaussian processes have been used by several authors spawning several variants.
Inearly accounts, Neal [1998], Williams and Rasmussen [1996] explore the integration over covariancehyperparameters using HMC in the regression setting.
Barber and Williams [1997] extend this to theclassification setting using HMC for sampling in the hyperparameter space and Laplace approximationfor the integrand over function values.
Murray and Adams [2010a] and Filippone et al.
[2013] focusedon MCMC schemes to sample covariance hyperparameters in conjunction with latent function values,mainly mitigating the coupling effect through reparameterisation.
Hensman et al.
[2015b] consideredjoint sampling of inducing variables and hyperparameters from the optimal variational posteriordistribution while [Bui et al., 2018] consider inference schemes for fully Bayesian sparse GPs
ina streaming setting.
More recently, Rossi et al.
[2021] studied fully Bayesian sparse GPs using
SG-HMC.
Rossi et al.
[2021] modify the generative model by adding a prior over the inducing inputs,and peform inference using SG-HMC over both Z and θ .
We list the most recent works in Table 1.
Background
Let f ∼ GP (0 , k θ ) be a Gaussian process prior with kernel function k θ depending on hyperparameters θ .
We are given noisy observations y
=
( y n ) Nn =1 ⊆ R of f = ( f ( x n ))
Nn =1 atinput data X = ( x n ) Nn =1 ⊆ R D .
We consider likelihoods which factorise over the data, likep ( y | f ) =
(cid:81) N n =1 N ( y n | f n , σ 2 ) for regression or p ( y | f )
= (cid:81) N n =1 B
( y n | ϕ ( f n )) for binary classification where ϕ ( · ) =
(cid:82) x
−∞ N ( a | 0 , 1) d a denotes the probit link function and B ( y n | ϕ ( f n )) =
ϕ ( f n ) y n (1 − ϕ ( f n )) (1 − y n ) is the Bernoulli distribution.
We wish to compute the posterior p ( f | y , θ ) .
In this section, we recapitulate the canonical inducing variable approximation of p ( f | y , θ ) by Titsias[2009] and its extension to a Bayesian treatment of the hyperparameters.
3.1 Sparse variational inference in Gaussian processes
Following Titsias [2009], we consider a set of inducing variables u
= { f ( z m ) } Mm =1 ⊆ R at inducinginputs Z = { z m } Mm =1 , z m ∈ R d .
The complete generative model can then be factored as,
[Equation]
We approximate the posterior p ( f, u | y , θ ) with a variational distribution:
[Equation]
where q ( u ) is chosen to minimise the Kullback–Leibler divergence KL( q ( f | θ ) ∥ p ( f | y , θ )) .
Minimising this KL divergence corresponds to maximising the evidence lower bound [Matthews et al. ,2016], henceforth called the ELBO:
[Equation]
Because the ELBO still depends on q ( u ) , this bound is called uncollapsed .
Hensman et al.
[2013,2015a] let q ( u ) be a Gaussian, which is optimal if the likelihood is Gaussian [Titsias, 2009], approximate the expectation using Monte Carlo, and maximise the ELBO using stochastic optimisation.
Onthe other hand, if the likelihood is Gaussian, Titsias [2009] computes optimal form for q ( u ) directly:
[Equation]
Plugging q ∗ ( u | θ ) back into Equation (5), the resulting bound is called collapsed , because it now onlydepends on θ and Z .
The collapsed bound is the objective that Titsias [2009] proposes.
[Equation]
Using the collapsed bound, approximate ML-II the consists of finding,
[Equation]
Predictions at new functions values can be made in O ( M 2 ) after an initial cost of O ( NM 2 ) .
Undercertain assumptions on the data generating process, even when M ≪ N , the approximate posteriorclosely resembles the posterior, and Equation (6) is a provably accurate approximation to Equation (2)[Burt et al., 2020].
3.2 Bayesian treatment of hyperparameters and sparse methods
The extension of the sparse variational framework to a Bayesian treatment of the hyperparametershas been previously considered by Hensman et al.
[2015b].
Extend the generative model with a priorp ( θ ) over the hyperparameters θ , and let the variational approximation of the posterior p ( f, u , θ | y )be q ( f, u , θ ) = p ( f | u , θ ) q ( u , θ ) .
The analogue of Equation (5) is
ELBO
[Equation]
[Equation]
8) and the optimal form for q ( u , θ ) can again be determined:
[Equation]
The distribution q ∗ ( u , θ ) does not have a closed form, and for general likelihoods, Hensman et al.[2015b] propose to approximate the expectation in (9) with quadrature and to sample from q ∗ ( u , θ )using HMC.
While this approach is quite general, in the case of Gaussian regression, it vastlyincreases the dimensionality of the state space over which HMC must be run relative to HMC in GPR,since the u are sampled in addition to the θ .
This increases the cost of the procedure, and impact thesuccess of the sampler.
An alternative approach to approximately inferring hyperparameters is to assume a parametric formfor q ( u , θ ) and maximise Equation (8) with respect to the variational parameters.
Bui et al.
[2018]took such an approach, assuming that q ( u , θ ) =
q ( u ) q ( θ ) , with both distributions Gaussian.
Similarapproaches have been applied to variational inference in state-space modelling, sometimes leveragingthe optimal form of q ( u | θ ) discussed earlier.
[Table]
3.3 Making predictions
The predictive posterior distribution for unknown test inputs X ∗ integrates over the joint posterior,
[Equation]
(10) (where we have suppressed the conditioning over inputs X, X ∗ for brevity).
The inner integralsimplifies to (cid:82) p ( f ∗ | f , u , θ ) p ( f | u , θ ) d f = p ( f ∗ | u , θ ) .
We discuss the predictive posterior in suchmodels in section 4. [Table]
Fully Bayesian SGPR with HMC: Doubly collapsed formulation
In the previous section, we observed that a major drawback of the approach taken in Hensman et al.[2015b] is the need to sample u , which for high-dimensional inputs or in cases where many inducingpoints are needed could introduce thousands of additional variables to sample.
In this section, weleverage the optimal form of q ( u | θ ) derived in Titsias [2009] to alleviate this sampling problem.
4.1 Collapsing the evidence lower bound (again)
We first derive the lower bound for this formulation and provide pseudo-code for the algorithm in
Algorithm 1.
As in the previous section, we wish to minimize the KL divergence between the trueand approximate posterior, KL( q ( f , u , θ ) || p ( f , u , θ | y ) .
Subtracting from the marginal likelihoodand rearranging,
[Equation]
) where we assign L θ ,Z = log M ( θ,z ) , which is defined in Equation (6).
In Equation (12), we substitutein the optimal marginal distribution q ∗ ( u | θ ) derived in Titsias [2009].
We can interpret the RHS of Equation (12) as a negative KL divergence as long as we account for anormalisation constant C for the un-normalised numerator term M ( θ,Z ) p ( θ ) .
Hence, we can re-writethe ELBO (in Equation (12)) as,
[Equation]
) where, q ∗ ( θ ) =
M ( θ,Z ) p ( θ ) /C .
Crucially, we can just write down the optimal variational distributionover θ (up to the normalising constant C ) by inspecting Equation (12) 2 . Further, by sampling directly we from q ∗ ( θ ) using MCMC we eliminate the need to sample the variables u .
By setting q ( θ ) =
q ∗ ( θ ) ,we find the doubly collapsed ELBO L ∗∗ = log C .
Although the value of L ∗∗ is computationallyintractable, given samples ( θ j ) Jj =1 from q ∗ ( θ ) , gradients of L ∗∗ with respect to variational parametersϕ can be estimated using the stochastic estimate of the canonical ELBO Equation (6): using the chainrule (showing explicit dependence of L ∗ and L ∗∗ on ϕ ),
[Equation]
where δδq L ∗ ϕ ( q ) is the functional derivative of L ∗ ϕ with respect to q , which is zero at q = q ∗ , becauseq ∗ optimises
L ∗ ϕ (it is a critical point, so the derivative is zero).
Further, the partial derivative of L ∗ ϕwith respect to variational parameters ( ϕ ) concerns just the first term of the LHS of Equation (12) asthe KL term KL( q ( θ )
|| p ( θ )) is independent of the variational parameters ( Z in our case).
4.2 Performing approximate inference
We deploy HMC to sample from the optimal variational posterior q ∗ ( θ ) along with optimising theinducing inputs Z in a hybrid scheme.
We alternate between the two steps allocating longer intervalsfor optimising Z for every HMC sampling run for the hyperparameters.
We note that this hybridscheme is much more computationally efficient than sampling u and θ jointly where one has totackle the coupling between inducing variables and hyperparameters in joint space.
Further, jointsampling is only feasible for moderate number of inducing variables while this scheme can scale tomuch larger datasets as the efficiency of sampling in the hyperparameter space is only dependent onthe dimensionality of the hyperparameter space rather than the number of inducing variables.
Theentire inference scheme is summarized in Algorithm 1.
1: Input: ELBO objective L θ, Z := L ( θ , Z ) (Equation (6)), gradient based optimiser optim() 2: 3: procedure W ARM -S TART 4: for fixed number of steps do 5:
Gradient step: Z, θ ←− optim ( L ( θ , Z ) ) 6: return initial values Z init and θ init 7:8: procedure T RAIN 9: ## Initialisation protocol 10: ▶ Initialise L θ,Z at warm-start values L ( θ init , Z init ) , lets call this ˆ L 11: ▶ Freeze kernel hyperparameters in the ELBO objective by setting requires grad= False . 12: 13: while not converged do 14: for t = 1 . . . T do ## start of training loop 15:16: • Gradient step: Z opt ←− optim (
Equation (14) shows the validity of taking the derivative of the stochastic ELBO) 17: if t mod L == 0 then 18: (## For every L gradient steps) 19: 20: • Draw J samples from the optimal hyperparameter variational distribution 21: log q ∗ ( θ ) ∝ L ( θ , Z opt ) + log p ( θ ) keeping Z opt fixed. ( θ j , p j ) HMC ←−−− H ( θ , p ) , (where H is the Hamiltonian) 22: where p denotes the zero-mean
auxilliary momentum variable in phase space with 23: the
same
dimensionality as
θ
• Compute stochastic ELBO
ˆ L = 1 J (cid:80) Jj =
, where θ
25: return Z
{ θ } posterior It is ultimately the posterior predictive (PP) distribution that is of interest rather than point estimatesor the hyperparameter posterior.
The Bayesian sparse GP predictive posterior entails integrating outthe posterior over inducing variables u and hyperparameters θ .
Once we have performed inference,we can approximate this directly,
[Equation]
where q ∗ ( u | θ , y ) =
N ( m ∗ , S ∗ ) is the optimal Gaussian variational distribution and
A =
K ∗ m K − 1 mmunder the SGPR scheme (Section 4) is available in closed form with m ∗ =
σ − 2 ( K mm +σ − 2 K mn K nm ) and S ∗ =
K mm ( K mm + σ − 2 K mn K nm ) − 1 K mm .
In either case, the internalintegral with respect to the inducing variables is analytic and the outer integral can be estimatedusing the samples collected from HMC to perform Monte Carlo estimation.
This yields a mixture of Gaussians,
[Equation]
[Equation] where J samples are (approximately) drawn from q ∗ ( θ ) via HMC.
The distribution inside thesummation is the Gaussian posterior predictive distribution for fixed hyperparameters with identicalmixing proportions.
The compute cost for the predictive posterior scales the sparse GPR cost linearlyin the number of samples.
The Monte Carlo approximation costs O ( JNM 2 ) for M inducing pointsand J hyperparameter samples.
Experiments
In the previous section, we discussed a hybrid scheme which leverages MCMC within the variationalsparse inducing variable formulation leading to fully Bayesian sparse Gaussian processes.
In theexperiments we demonstrate the feasibility of this scheme relative to several benchmarks and assessregression performance on a 1-dimensional illustrative example and a range of other datasets.
We also compare with exact GPs where the cost of using a gradient based sampler is prohibitive foreven moderately large datasets requiring several inversions of the full covariance matrix.
We showthat using the doubly collapsed scheme proposed in this work is a much more attractive alternativefor large datasets as compared to direct HMC in GPR, and sampling is more efficient than in theuncollapsed bound used in Hensman et al.
[2015b].
We henceforth refer to benchmark methods as follows: SGPR + HMC refers to Bayesian GPswith doubly collapsed variational inference with NUTS, as described in section 3 (compatiblewith Gaussian likelihoods); we benchmark this model against sparse GPs ( SGPR )
[Titsias, 2009]and Stochastic Variational GPs ( SVGP ) both using approximate ML-II [Hensman et al., 2015a].
Additionally, we consider the sparse, joint sampling inference scheme proposed in [Hensman et al.
,2015b] which gives a natural benchmark.
We call this model JointHMC .
The FBGP method inspiredby Rossi et al.
[2021] uses NUTS to sample from the posterior over ( Z, θ ) .
We also present analysis where we fix inducing point locations at a random subset of the training data(as opposed to interleaving as per Algorithm 1) and only learn hyperparameters using NUTS.
Weprovide several details about the experimental set-up in the supplementary.
5.1 One dimensional synthetic data
Method
SGPR SGPR + HMC JointHMC RMSE 0.
NLPD 0.
We sample noisy observations from f ( x ) =sin(3 x ) + 0 .
cos( πx ) with the constraint ( x <− 2) and ( x > 2) .
Figure 2 shows the results for
SGPR along with the fully Bayesian schemes.
We keep data split and noise identical across thethree models to facilitate a comparison.
Whilethere is significant data to identify the hyperparameters we notice that the models mainly differin their extrapolation abilities away from thetraining data.
SGPR with ML-II overfits to thetraining data and recovers a low noise solution while the SGPR + HMC scheme recovers a moremoderate fit and performs significantly better in terms of RMSE and NLPD on unseen data Table 3.
We note that the JointHMC scheme which samples both ( u , θ ) overfits in the central missing dataregion.
We use M = 25 inducing locations across all methods which are optimised according to theprotocol of each method and recover a very similar spatial distribution.
5.2 UCI regression benchmarks
We compare our approach across methods on 5 standard small to medium-sized UCI benchmarkdatasets.
Following common practice, we use a 20 % randomly selected held out test-set [Rossi et al. ,2021, Havasi et al., 2018] and scale the inputs and outputs to zero mean and unit standard deviationwithin the training set (we restore the output scaling for evaluation) [Salimbeni and Deisenroth, 2017].
While we could use any kernel, we choose the RBF-ARD kernel with a lengthscale for each dimension.
For consistency we initialise all the inducing locations ( Z ) identically across the methods, i.e. byusing the same random subset of training data split.
We note that adapting the inducing locations [Table] brings serious gains in prediction performance versus keeping them fixed (Figure 3).
Further, the JointHMC scheme underperforms SGPR (with ML-II) and SGPR + HMC.
This is not surprisinggiven that the JointHMC bound Equation (9) does not incorporate the optimal setting for q ( u ) andwas originally motivated by the need for a fully Bayesian scheme for generalised likelihoods.
Themethod SGPR + HMC significantly improves upon JointHMC in the specific Gaussian likelihoodcase.
5.3 Ablation study
In order to understand the efficacy of Algorithm 1 we conduct an ablation study where we performinference in the same manner, but keeping inducing locations fixed.
Algorithmically, this impliesthat we don’t need to compute the stochastic ELBO ˆ L and just conduct a single sampling run for thehyperparameters.
The results across 10 splits are summarised in Table 5.
Sensitivity to M : We benchmark SGPR+HMCand JointHMC on the Elevator dataset ( N =16599 , D = 18 ) which demands a larger M .
SGPR+HMC outperforms JointHMC for thisdataset across different M but the advantage ismore pronounced at smaller M .
Further, ourmethod took 1248 vs. 2109 wall clock sec.
forthe joint scheme for the same number of hyperparameter samples and 500 inducing points.
[Table]
5.4 Runtimes
While it is possible to train exact GPR with HMC for datasets of this size (in terms of N ) it is important to lookat the trade-off in terms of compute cost.
In Figure 4 werecord the average number of wall clock seconds to draw500 samples under each method.
The cost of samplingis O ( N ) for SGPR + HMC but O ( N 3 ) for Exact GPR + HMC.
JointHMC deals with a higher dimensional phasespace ( u , θ ) hence requires more tuning.
We don’t includetuning time for a fair comparison.
Discussion
The evidence framework continues to be the pre-dominantmethod for training Gaussian processes since their inception into modern machine learning [Rasmussen and Williams, 2006].
While the marginal likelihood is a compelling model selection objective as it offers an inherenttrade-off between data-fit and complexity, it is susceptibleto overfitting and other pathologies leading to biased inference.
This work investigates sparse Gaussian processesrelying on inducing variables in conjunction with Bayesianhyperparameter inference.
Bayesian hyperparameter inference in GPs is however intractable and one has to consider balancing between objectives of computational cost, predictionaccuracy and robustness of uncertainty intervals.
While in straightforward conditions the fully Bayesian approach might be counter-productive, most real-world applications of GPs rely on engineering sophisticated hand-crafted kernels involving many hyperparameters where there risk ofoverfitting is pronounced and further, harder to detect.
A more robust solution is to incorporateprediction intervals that reflect these uncertainties in the model choice.
Studying full Bayesianinference in more sophisticated GP models like deep [Damianou and Lawrence, 2013], warped[Snelson et al., 2004] and convolutional GPs [Van der Wilk et al., 2017] will offer greater insight tothis question and is an imminent direction of future work.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] We provide extensive background to justify.
(b) Did you describe the limitations of your work?
[Yes] We include a brief discussion in the supplementary.
(c) Did you discuss any potential negative societal impacts of your work?
[No] This work
is largely methodological but we touch upon this briefly in the limitations section inthe supplementary.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] We providederivations with discuss assumptions in section Section
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper-
imental results (either in the supplemental material or as a URL)?
[Yes] We includeassumptions and other experimental details like the prior hyperparameters, learning
rate and data splits in the supplementary.
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes] We include these details along with a pymc3 code snippet for
reproducibility.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] We report std error of the mean across 10 splits
for
mostexperiments.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs , internal cluster or cloud provider)
[Yes] Since we mainly use small tomoderate sized datasets we
conducted most experiments on the CPU.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots
, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]