Efficient identification of informative features in simulation-based inference
Abstract
Simulation-based Bayesian inference (SBI) can be used to estimate the parametersof complex mechanistic models given observed model outputs without requiringaccess to explicit likelihood evaluations.
A prime example for the application of
SBI in neuroscience involves estimating the parameters governing the responsedynamics of Hodgkin-Huxley (HH) models from electrophysiological measurements, by inferring a posterior over the parameters that is consistent with a set ofobservations.
To this end, many SBI methods employ a set of summary statistics orscientifically interpretable features to estimate a surrogate likelihood or posterior.
However, currently, there is no way to identify how much each summary statisticor feature contributes to reducing posterior uncertainty.
To address this challenge,one could simply compare the posteriors with and without a given feature includedin the inference process.
However, for large or nested feature sets, this wouldnecessitate repeatedly estimating the posterior, which is computationally expensiveor even prohibitive.
Here, we provide a more efficient approach based on the SBImethod neural likelihood estimation (NLE): We show that one can marginalizethe trained surrogate likelihood post-hoc before inferring the posterior to assessthe contribution of a feature.
We demonstrate the usefulness of our method byidentifying the most important features for inferring parameters of an example HHneuron model.
Beyond neuroscience, our method is generally applicable to SBIworkflows that rely on data features for inference used in other scientific fields.
Introduction
Mechanistic models are an elegant way to encode scientific knowledge about the world in the formof numerical simulations.
They include models such as Kepler’s laws of planetary motion [1], the
SEIR model [2] for describing the spread of infectious diseases or the Hodgkin-Huxley (HH) modelfor the dynamics of action potentials in neurons [3].
Efficiently constraining the parameters ofsuch models by measurements is a key problem in many disciplines [4–7].
Since these modelsgive rise to intractable likelihood functions, however, classical likelihood-based Bayesian methodssuch as variational inference [8] or Markov Chain Monte Carlo [9] cannot be used directly.
Thereare algorithms, such as ABC-MCMC [10] or pseudo-marginal methods [11] that deal with thisproblem, however, they have potentially slow convergence rates and are computationally expensive.
To overcome this problem, several techniques collectively known as simulation-based inference (SBI)have recently been developed [12].
Leveraging the ability to simulate the model, these techniquesobtain estimates of the likelihood or posterior from simulated data.
The most recent of thesealgorithms such as neural likelihood estimation (NLE) [13] or neural posterior estimation (NPE) [14– p r obab ili t y feature(marginal) likelihood f ea t u r efeaturep r obab ili t y featureposterior pa r a m e t e r parameterpa r a m e t e r parameterpa r a m e t e r parameter
MDN prior p r obab ili t y pa r a m e t e r parametersimulated data m V ms
[Figure]
observation
[Figure]
16] employ state-of-the-art neural density estimators to learn tractable surrogates of these functionsto be evaluated instead of the real quantities.
To this end, summary statistics or features capturingessential aspects of the model’s dynamics are defined by the scientist to reduce the high dimensionalmodel output to manageable scale, in turn decreasing the problem complexity and computation costs.
This is often done by hand to emphasize specific aspects of the data or to aid scientific interpretation[12], but can also be automated [17–23].
In neuroscience, many different approaches have been developed in order to find suitable parametersof models of neural activity [4, 5, 24–29].
SBI approaches have been used to infer parameters inbiophysical neuron models from measurements of neural activity in a Bayesian way [15, 30, 31].
Forinference of HH models from electrophysiological data, features such as action potential thresholdor width and resting membrane potential or spike count can be used, reflecting measures thatelectrophysiologists use to characterize recorded neurons.
For scientific interpretation of SBI results, it is often of interest which features, or combinations offeatures, have the biggest impact on the posterior and which parameters they affect specifically.
Tothis end, one could compare the posterior uncertainty estimated with and without including a specificfeature in the SBI method — the increase in uncertainty resulting from not relying on that featurecan be used as a measure of its importance and on average is equivalent to its mutual information.
Of course, this approach can also be applied to whole subsets of features.
To evaluate an entire setof features exhaustively would require re-estimating the posterior many times (Fig. 1, grey), whichwould scale prohibitively with the number of features.
To address this issue, we here introduce amethod called Feature Selection Through Likelihood Marginalization (FSLM) to compute posteriorsfor arbitrary subsets of features, without the need for repeated training (Fig. 1, black).
To achievethis, we here use NLE [32] and exploit the marignalization properties of mixture density networks(MDNs)
[33], which can be used with NLE as a density estimator: instead of re-estimating thesurrogate likelihood from scratch, we marginalize it analytically with respect to a given (set of)features before applying Bayes rule to obtain the posterior estimate.
This way, we can efficientlycompare the posterior uncertainty with and without including a certain feature.
For a simple linear Gaussian model and non-linear HH models, we show that the obtained posteriorestimates are as accurate and robust as when repeatedly retraining density estimators from scratch,despite being much faster to compute.
We then apply our algorithm to study which features are themost useful for constraining posteriors over parameters of the HH model.
Our tool allows us also tostudy which model parameter is affected by which summary feature.
Finally, we suggest a greedyfeature selection strategy to select useful features from a predefined set.
Methods
2.1 Neural Likelihood Estimation (NLE)
Neural Likelihood Estimation [32] is a SBI method which approximates the likelihood p ( x | θ ) ofthe data x given the model parameters θ by training a conditional neural density estimator on datagenerated from simulations of a mechanistic model.
Using Bayes rule, the approximate likelihoodcan then be used to obtain an estimate of the posterior (Fig. 1).
Unlike NPE [15, 16], NLE requiresan additional sampling or inference step [34] to obtain the posterior distribution.
However, it allowsaccess to the intermediate likelihood approximation, a property we will later exploit to develop ourefficient feature selection algorithm
(see Sec. 2.3).
First, a set of N parameters are sampled from the prior distribution θ n ∼ p ( θ ) , n ∈ { 1 , . . .
, N } .
With these parameters, the simulator is run to implicitly sample from the model’s likelihood functionaccording to x n ∼ p ( x | θ n ) .
Here, x n = ( x 1 , . . .
, x N f ) are feature vectors that are usually takento be a function of the simulator output s with the individual features x i
=
f
i ( s ) , where i ∈{ 1 , . . . , N f } , rather than the output directly.
For mechanistic models whose output are time series, falso reduces the dimensionality of the data to a set of lower dimensional data features.
The resultingtraining data { x n , θ n } 1: N ∼ p ( x , θ ) can then be used to train a conditional density estimatorq ϕ
( x | θ ) , parameterized by ϕ , to approximate the likelihood function.
Here, we use a Mixture
Density Network for this task [33], the output density of which is parameterised by a mixture of
Gaussians, which can be marginalized analytically.
Thus, ˆ p ( x | θ ) =
(cid:80) k π k N ( µ k , Σ k ) with theparameters ( µ k , Σ k , π ) being non-linear functions of the inputs θ and the network parameters ϕ .
The parameters ϕ are optimized by maximizing the log-likelihood L ( x , θ )
= 1 N (cid:80) n log q ϕ ( x | θ )over the training data with respect to ϕ .
As the number of training samples goes to infinity, thisis equivalent to maximizing the negative Kullback-Leibler (KL) divergence between the true andapproximate posterior for every x ∈ supp ( x ) [32]:
[Equation]
After obtaining such a tractable likelihood surrogate, Bayes rule can be used to obtain an estimateof the posterior conditioned on an observation x o (see Eq. 2) for instance via Markov Chain Monte
Carlo (MCMC):
[Equation]
2.2 A naive algorithm for quantifying feature importance
Given this posterior estimate, we would now like to answer the following question: for which featurex i from a vector of features x does the uncertainty of the posterior estimate
ˆ p ( θ | x ) increase themost, when it is ignored?
For a single feature, a naive and costly algorithm does the following: iterateover x to obtain x \ i =
( x 1 , . . .
x i − 1 , x i +1 , . . .
, x N ) , train a total of N+1 density estimators to obtainthe likelihoods ˆ p ( x \ i | θ ) and ˆ p ( x | θ ) , sample their associated posteriors and compare ˆ p ( θ | x \ i ) to thereference posterior ˆ p ( θ | x ) with every feature present.
The same procedure can also be applied toquantify the contribution of any arbitrary subset of features.
As this procedure requires estimating thelikelihood based on the reduced feature set from scratch, for each feature (set) (Fig. 1, grey arrows),it is computationally costly.
To quantify the contribution of different features more efficiently, wewould thus need a way to avoid re-estimating the posterior with each feature left out.
- 5 θ5 θ5 θp( θ x ,x ,x ,x ) ∣ 0 1
Ground truth NLE+MDNFSLM θ o a5 θ5 θ5 θp( θ x ,x ,x ) ∣ 1 2 3 b θθθ
Ground truth x 0 x 1 x 2 xθθθ
FSLM
I I Q Q R R ( ( p p) )
̂
̂ \i
Removed feature
M ode l pa r a m e t e r s
Removed feature
M ode l pa r a m e t e r s c
2.3 Efficient quantification of feature importance through post-hoc likelihood marginalization
To more efficiently estimate posteriors for any subset of features, we propose to marginalize thelikelihood estimate obtained via NLE post-hoc, as opposed to training a separate density estimatorfor every new set of features to obtain the marginal estimates (Sec. 2.2, Fig. 1).
We refer to thisalgorithm as efficient Feature Selection through Likelihood Marginalization (FSLM).
Suppose we want to evaluate the joint increase in posterior uncertainty for a subset of features.
Wepartition our feature vector x = [ x 1 , x 2 ] , such that x 2 contains the features to be removed from theinference.
To avoid the need for training the surrogate likelihood q ϕ for such a feature subset, we canrewrite the posterior with respect to x 1 as
[Equation]
Since we parameterize the approximate likelihood q ϕ as an MDN, we can perform this marginalizationanalytically: For a K-component MDN, the parameters of each mixture component can be similarlypartitioned to x [35, Chapter 2.3.2]:
[Equation]
Then the marginalization with respect to x 2 results in the following posterior distribution
[Equation]
where π = π ( θ ) , µ = µ ( θ ) and Σ = Σ ( θ ) .
FSLM can thus sample from the posterior givenarbitrary feature subsets without estimating the surrogate likelihood q ϕ from scratch.
We implement FSLM using python 3.8, building on top of the public sbi library [36].
All the code isavailable at github.com/mnwncqvots/fslm.
All computations were done on an internal cluster running
Intel(R)
Xeon(R)
Gold 6226R CPUs @ 2.90GHz.
2.4 Measures of posterior uncertainty
Assuming the NLE procedure converges and the density estimator is sufficiently flexible, differencesin posterior uncertainty can be ascribed to differences in information content of features or noise in thedata.
Given that that noise in the data remains constant, relative differences in posterior uncertainty [Table] can be used to assess the contributions of individual summary features.
Hence, we can identifyinformative summary statistics by quantifying the change in uncertainty of ˆ p ( θ | x \ i ) with respect toˆ p ( θ | x ) .
For this we use two metrics:
Ratio of IQR of the marginals: We use the ratio of inter quantile ranges (IQR) of the 1D-marginalsof ˆ p ( θ | x 1 ) and ˆ p ( θ | x ) .
This lets us deduce if specific features x i reduce the uncertainty of someparameters θ j more than others.
We use IQRs as a measure of spread as it is much less susceptible tooutliers compared to the standard deviation.
The resulting visualization allows to precisely pin downwhich parameter is affected by which summary feature.
Non-parametric estimate of KL divergence: As a second metric, we use a non-parametricestimate of the KL divergence between ˆ p ( θ | x 1 ) and ˆ p ( θ | x ) .
Since a posterior estimate, onaverage, cannot become more constrained if features are removed, increases in the KL estimateindicate an increase in uncertainty overall.
To estimate the KL-divergence between ˆ p ( θ | x 1 ) andˆ p ( θ | x ) , we employ a purely sampled based estimator [37, 38].
This is necessary because NLE onlyallows to evaluate unnormalized posteriors which circumvents having to evaluate them explicitly.
The estimator makes use of 1-nearest neighbor search and is implemented using k-d trees [39], toreduce the 2 N comparison operations to a manageable number.
With samples X =
{ X i } Ni =1 and Y =
{ Y i } Mi =1 from ˆ p ( θ | x 1 ) and ˆ p ( θ | x ) respectively, the KL-divergence KL (ˆ p ( θ | x 1 )
|| ˆ p ( θ | x ))is estimated as
[Equation]
where d is the dimensionality of the samples, and || · || denotes the ℓ 2 -norm.
The Maximum Mean
Discrepancy (MMD) [40, 41] is an alternative measure that could be used here.
We performedpreliminary experiments in which the KL estimate was more consistent than the MMD.
Results
3.1 Example 1: Linear Gaussian model
We illustrate our algorithm in a simple toy problem, where by construction, we know which featuresshould have which effect.
We used a Linear Gaussian Model (LGM) with a three dimensionalparameter space θ =
[ θ 0 , θ 1 , θ 2 ] and four dimensional feature space x =
[ x 0 , x 1 , x 2 , x 3 ] .
θ isdrawn from a uniform prior θ i ∼ U ( − 5 , 5) , i ∈ { 0 , 1 , 2 } , with the features x being noisy linear [Figure] transformations of the parameter vector θ .
The mean µ ( θ ) and covariance Σ are defined byµ ( θ ) =
µ 0 + Lθ , Σ
= σ 2 1 ,
L =   1 00 10 10 0  .
With scale σ and shift µ 0 .
We construct L such that we can easily verify that the posterior reacts toand interacts with marginalized features as expected.
In our example, we let feature x 0 solely dependon θ 0 , x 1 only on θ 1 , x 2 on both θ 1 and θ 2 , and x 3 without influence on any model parameter.
The resulting posterior is well concentrated on θ o and the NLE+MDN estimate matches the analytically computed posterior well (Fig.
2 a ).
To show that FSLM can correctly identify individualfeature importance, while being more efficient than re-running NLE for each subset, we comparedsamples from the FSLM-derived posterior estimates for every three feature subset to those obtainedusing the analytical marginal likelihood p ( x \ i | θ ) , i ∈ { 0 , 1 , 2 , 3 } and brute-force NLE estimatesusing separate q ϕ ( x \ i | θ ) for each subset (Fig.
2 b , Appendix Fig.5 a - c ).
The brute-force baselinewas conducted with both a MDN and a Masked Autoregressive Flow (MAF) [42]. The MDNs hadthree hidden layers and 10 mixture components. The MAF consisted of five 3 layer deep MADEs[43].
Training was done on 10,000 samples, then 500 samples were drawn from each posterior, for atotal of 10 different initializations.
Training was terminated when the log-likelihoods had stoppedimproving on a validation set that was split from the training set at 10% over 20 epochs.
The posterior estimates obtained from FSLM matched those from the analytic ground truth veryaccurately (Fig.
2 b - e , Appendix Fig. 5 a - c and Tab. 1), while being several times faster to computethan both NLE alternatives (see Tab. 1).
As expected from the construction of our model, weobserved that ˆ p ( θ | x \ 0 ) was not only less constrained compared to p ( θ | x ) (Fig.
2 a ), but thatˆ p ( θ 0 | x \ 0 )
=
p ( θ 0 )
=
U ( − 5 , 5) .
We also found that FSLM correctly identified the dependencystructure between the other parameters and features (Appendix Fig.
5 a - c ), with FSLM matching theanalytic ground truth while being much faster than both NLE approaches (Tab. 1).
3.2 Example 2: Hodgkin-Huxley neuron model
Next, we turned to a more realistic complex mechanistic model used for simulations of neurons.
Forour experiments we used a single compartment Hodgkin-Huxley model [3], Eq. 8, derived from
Pospischil et al.
In addition to the standard sodium, potassium and leak currents to generatespiking this model also considers a slow non-inactivating K + current to model spike-frequencyadaptation and a high-threshold Ca 2+ current to generate bursting. [Equation]
[Equation]
Eq.
8 models the evolution of the membrane voltage V t
=
V ( t ) given a stimulus I ( t )
=
I t .¯
g i , i ∈
{ Na, K, l, M, L } are the maximum conductances of the sodium, potassium, leak, adaptivepotassium and calcium ion channels, respectively, and E i are the associated reversal potentials.
I tdenotes the current per unit area, C the membrane capacitance, A Soma the compartment area andn, m, h, q, r and p represent the fraction of independent gates in the open state, based on Hodgkinand Huxley [3].
For the remaining equations and parameter settings, see Appendix A.3.
We then compared the performance of FSLM and brute-force NLE to measure how commonly usedsummary features constrain the posterior of an exemplary voltage trace.
Specifically, we used a setof 10 summary statistics commonly of interest to neurophysiologists, which capture the responsedynamics of the HH model and can be used to differentiate cortical cell types.
They are brieflyexplained in Table 3.
To extract these features from the simulated traces we use the Allen SDK.
To train the density estimator we simulated 1 million observations with parameters drawn from a Uniform prior over biologically plausible parameter ranges (see A.3).
However, because a substantialfraction of the generated observations did not produce spiking behavior, some features such as spikerate were not defined.
Hence, these observations could not be used to train the density estimator.
Weemployed two different techniques introduced by Lueckmann et al.
[15] to reduce the amount ofinvalid samples returned by the prior and to compensate for any remaining invalid observations inthe training data (for details see Appendix A.2).
This yielded accurate posterior estimates, despiteincomplete data.
The change in uncertainty was then assessed on 3000 samples from the MDNestimates of ˆ p ( θ | x ) and ˆ p ( θ | x \ i ) .
{+Architectures and training were kept the same as in Sec.3.
+}
We found that both NLE+MDN and FSLM identified the same two features as yielding the highestincrease in posterior uncertainty (Fig.
3 a ): the action potential threshold (APT) and the action potential count (APC).
Specifically, APT strongly affected the threshold voltage V T , the leak conductanceg leak as well as the sodium conductance g Na .
APC, on the other hand, influenced the leak reversalpotential E leak , the leak conductance g leak , as well as C and g Na (Fig. 3 a ).
In contrast, the actionpotential amplitude (APA), as well as other features, did not have a sizable impact on the posterior inthis example (Fig. 3 a ). These results were independent of whether they were obtained by brute-force NLE+MDN, or FSLM, however FSLM was substantially faster ( ≈ 5 -fold speedup, Table 1).
Thisspeedup is even more significant compared to NLE+MAF, while being almost as accurate, whenmeasured against a common baseline.
As a consequence, simulations based on samples from ˆ p ( θ | x ) and ˆ p ( θ | x \ APA ) were comparableto the observed voltage trace V o ( t ) , while the sample drawn from ˆ p ( θ | x \ APT ) showed different a [Figure] Var(V ) m μ ( [Figure]
[Figure] APC latency APA adapt APW
APA
AHP
APT μ (V ) rest
F ea t u r e 1 st 2 nd 3 rd 4 th 5 th 6 th 7 th 8 th 9 th 10 th
S e l e c t ed a s b spiking behavior, with much longer inter spike intervals in the simulation (Fig. 3 b ).
This resultedfrom higher values for the sodium conductance g Na and threshold voltage V T , compared to θ o , forthe particular simulation shown (Fig.
3 a , c ).
The measured increase in uncertainty is also directlyvisible in the two-dimensional marginal distributions of the posterior, where the removal of APTlead to a much less constrained posterior estimate (Fig.3 c ), while the removal of APA did not affectthe posterior uncertainty much (Fig.3 d ).
Also, the measured summary statistics from samples fromˆ p ( θ | x \ APT ) and ˆ p ( θ | x \ APA ) showed the expected behavior: the APT feature was much morevariable in samples from ˆ p ( θ | x \ APT ) than in ˆ p ( θ | x ) , while the estimated value for APA did notvary as drastically even for samples from ˆ p ( θ | x \ APA )
(Fig. 3 e ).
3.3 Efficient greedy feature selection using FSLM
In the previous section, we showed that FSLM can be used to study how the removal of individualfeatures increases the uncertainty of the posterior.
Next we show how FSLM can be used to studyhow the addition of features decreases uncertainty and provide a method to efficiently search forinformative features by greedy selection of new features.
First, we used NLE to obtain a posterior estimate given a candidate set of features.
We then used
FSLM to obtain all the 1D marginal likelihoods, sampled their associated posteriors
ˆ p ( θ | x i ) andcomputed their KL divergences to the full posterior.
We subsequently selected the feature x i thatminimizes the KL, before we repeated the procedure for all 2D marginals
ˆ p ( θ | ( x i , x j )) whichinclude the selected feature x i .
We can iterate this procedure until a desired number of features isidentified.
This is equivalent to a beam-search [45] with a beam size equal to one.
To investigatecombinations of features the scheme is easily extensible to larger beam sizes.
To demonstrate this, we turned again to the HH example already studied in Sec. 3.2.
We used 1million samples for training and sampled every posterior 1000 times.
To check for consistency, werepeated the procedure 50 times.
Compared to selecting features at random, selecting features basedon the results of our greedy algorithm consistently yielded more constrained posterior estimates (see
Fig.
4 a ).
Furthermore, we found the three most informative features to be µ ( V rest ) , APT , AHP ,respectively, which were consistently chosen in this order (see Fig.
4 b ).
Discussion
We set out to develop an efficient method to characterize the relative contribution of different featuresto constraining the posterior in simulation-based inference.
To this end, we used NLE with a mixturedensity network to estimate the surrogate likelihood and marginalize it analytically.
We note that the same approach can also be implemented in Approximate Bayesian Computation (ABC) [46, 47], buttypically neural density estimation approaches significantly outperform ABC approaches Lueckmannet al.
We showed that our method, FSLM, correctly identifies influential features both in asimple linear Gaussian model, as well as in a complex mechanistic model, namely the HH modelin neuroscience.
In both cases, FSLM, was as accurate as repeated application of NLE, but manytimes faster.
FSLM can thus provide guidance on how to specify, prepare and select appropriatesummary statistics, while also allowing to study which parameters are constrained by which featuresto gain insight into the scientific structure of the model.
In contrast to methods developed forautomatically deriving optimal (combinations of) summary features [17–19, 21, 23], FSLM focuseson making the evaluation and selection of individual scientifically relevant feature sets interpretable.
Our visualization method allowed us to identify which features influenced the posterior uncertaintyover which model parameters, and for which model parameters the posterior was not influencedby any feature.
Compared to approaches that aim to directly quantify the mutual information (MI)between sets of parameters and features [23], our method aims to quantify the relationship betweenmodel parameters and features for individual observations.
Computing estimates of the MI forsimulator models is difficult without access to a closed form likelihood and purely sample basedestimators [49, 50] do not allow for inspection of individual or small sets of observations.
However,this is of interest when different cells, cell types or cell families are constrained by different featuresets.
Nonetheless, FSLM still retains the option to obtain a single ranking of features across a wholedataset by averaging over all observations in the set, which would then be equivalent to the MIbetween posterior and prior (the trained neural network is amortized and, thus, requires no retrainingfor different observations).
For the case of a Hodgkin-Huxley model frequently used in neuroscience as a mechanistic modelof neural activity, we showed that action potential threshold and action potential count were themost informative features, jointly constraining the action potential threshold, the resting potential,membrane capacitance as well as model conductances (except for that of potassium).
Removingthese features from the inference procedure led to much more poorly constrained posteriors andsimulations that did not match the original trace.
We note some limitations of FSLM and the metrics introduced: First, the inherent dependence on
MDNs for the conditional neural density estimation limits overall accuracy for highly complexposteriors or likelihoods, compared to e.g. normalizing flows, as it only works well for likelihoodsthat are well approximated by mixtures of Gaussians.
Additionally, the use of full covariance matricescan be problematic in higher-dimensional feature spaces due to the curse of dimensionality and onewould have to evaluate whether the use of e.g. diagonal covariances is more suitable.
Second, sinceour metrics are relative to the overall level of uncertainty in a given posterior marginal, for very sharpdistributions a small increase in variance can already lead to a large increase in our metrics, whichwould not be the case for a broad distribution.
While this may be the desired behavior in some cases,it might be misleading in others.
Third, the effect of a feature may be judged differently depending onwhether the effect of removing it from a large set of features is being assessed, or whether it is beingadded early in a greedy feature selection run.
Finally, the approximation quality of NLE, thus also
FSLM, depends crucially on the choice of prior [51].
However, posterior predictive checks [52, 53]and coverage tests [54] applicable to NLE can also be applied to FLSM to verify the correctness ofthe inference results.
Our method opens the door for careful studies on the relationship between data features and modeldistributions.
In neuroscience, for example, we could explore whether commonly used features suchas those used here constrain the posterior equally across different cellular families or types — forexample, is action potential threshold equally important for modeling fast-spiking interneurons andfor cellular families with more complex firing behavior, or are additional features needed in thiscase?
Given the efficient exploration of the relationship between features and posterior uncertainty,we believe that our method will also be applicable to a wide range of inference problems beyondneuroscience.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Sec. 4.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] We donot see negative social impact of our work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See Sec.2.3.(b)
Did you specify all the training details (e g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Sec.
3.1 and 3.2.
(c) Did you report error bars (e g., with respect to the random seed after running experi- ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Sec. 2.3.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
[Table]
using/curating?
[Yes] No data from third parties was used.
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
[Table]
information or offensive content?
[Yes]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A] [Table]