Efficient identification of informative features in simulation-based inference
Abstract
Simulation-based Bayesian inference (SBI) can be used to estimate the parametersof complex mechanistic models given observed model outputs without requiringaccess to explicit likelihood evaluations.
A prime example for the application of
SBI in neuroscience involves estimating the parameters governing the responsedynamics of Hodgkin-Huxley (HH) models from electrophysiological measurements, by inferring a posterior over the parameters that is consistent with a set ofobservations.
To this end, many SBI methods employ a set of summary statistics orscientifically interpretable features to estimate a surrogate likelihood or posterior.
However, currently, there is no way to identify how much each summary statisticor feature contributes to reducing posterior uncertainty.
To address this challenge,one could simply compare the posteriors with and without a given feature includedin the inference process.
However, for large or nested feature sets, this wouldnecessitate repeatedly estimating the posterior, which is computationally expensiveor even prohibitive.
Here, we provide a more efficient approach based on the SBImethod neural likelihood estimation (NLE): We show that one can marginalizethe trained surrogate likelihood post-hoc before inferring the posterior to assessthe contribution of a feature.
We demonstrate the usefulness of our method byidentifying the most important features for inferring parameters of an example HHneuron model.
Beyond neuroscience, our method is generally applicable to SBIworkflows that rely on data features for inference used in other scientific fields.
Introduction
Mechanistic models are an elegant way to encode scientific knowledge about the world in the formof numerical simulations.
They include models such as Kepler’s laws of planetary motion [1], the
SEIR model [2] for describing the spread of infectious diseases or the Hodgkin-Huxley (HH) modelfor the dynamics of action potentials in neurons [3].
Efficiently constraining the parameters ofsuch models by measurements is a key problem in many disciplines [4–7].
Since these models giverise to intractable likelihood functions, however, classical likelihood-based Bayesian methods suchas variational inference [8] or Markov Chain Monte Carlo [9] cannot be used.
To overcome thisproblem, several techniques collectively known as simulation-based inference (SBI) have recentlybeen developed [10].
Leveraging the ability to simulate the model, these techniques obtain estimatesof the likelihood or posterior from simulated data.
The most recent of these algorithms such asneural likelihood estimation (NLE) or neural posterior estimation (NPE) employ state-of-the-artneural density estimators to learn tractable surrogates of these functions to be evaluated instead ofthe real quantities [11–15]. To this end, summary statistics or features capturing essential aspects of p r obab ili t y feature(marginal) likelihood f ea t u r efeaturep r obab ili t y featureposterior pa r a m e t e r parameterpa r a m e t e r parameterpa r a m e t e r parameter
MDN prior p r obab ili t y pa r a m e t e r parametersimulated data m V
[Figure]
observation
[Figure]
the model’s dynamics are defined by the scientist to reduce the high dimensional model output tomanageable scale, in turn decreasing the problem complexity and computation costs.
This is oftendone by hand to emphasize specific aspects of the data or to aid scientific interpretation [10], but canalso be automated [16–19].
In neuroscience, many different approaches have been developed in order to find suitable parametersof models of neural activity [4, 5, 20–25].
SBI approaches have been used to infer parameters inbiophysical neuron models from measurements of neural activity in a Bayesian way [12, 26, 27].
Forinference of HH models from electrophysiological data, features such as action potential thresholdor width and resting membrane potential or spike count can be used, reflecting measures thatelectrophysiologists use to characterize recorded neurons.
For scientific interpretation of SBI results, it is often of interest which features, or combinations offeatures, have the biggest impact on the posterior and which parameters they affect specifically.
Tothis end, one could compare the posterior uncertainty estimated with and without including a specificfeature in the SBI method — the increase in uncertainty resulting from not relying on that featurecan be used as a measure of its importance and on average is equivalent to its mutual information.
Of course, this approach can also be applied to whole subsets of features.
To evaluate an entire setof features exhaustively would require re-estimating the posterior many times (Fig. 1, grey), whichwould scale prohibitively with the number of features.
To address this issue, we here introduce amethod called Feature Selection Through Likelihood Marginalization (FSLM) to compute posteriorsfor arbitrary subsets of features, without the need for repeated training (Fig. 1, black).
To achievethis, we here use NLE [28] and exploit the marignalization properties of mixture density networks(MDNs)
[29], which can be used with NLE as a density estimator: instead of re-estimating thesurrogate likelihood from scratch, we marginalize it analytically with respect to a given (set of)features before applying Bayes rule to obtain the posterior estimate.
This way, we can efficientlycompare the posterior uncertainty with and without including a certain feature.
For a simple linear Gaussian model and non-linear HH models, we show that the obtained posteriorestimates are as accurate and robust as when repeatedly retraining density estimators from scratch,despite being much faster to compute.
We then apply our algorithm to study which features are themost useful for constraining posteriors over parameters of the HH model.
Our tool allows us also tostudy which model parameter is affected by which summary feature.
Finally, we suggest a greedyfeature selection strategy to select useful features from a predefined set.
Methods2.1 Neural Likelihood Estimation (NLE)
Neural Likelihood Estimation [28] is a SBI method which approximates the likelihood p ( x | θ ) ofthe data x given the model parameters θ by training a conditional neural density estimator on datagenerated from simulations of a mechanistic model.
Using Bayes rule, the approximate likelihoodcan then be used to obtain an estimate of the posterior (Fig. 1).
Unlike NPE [12, 13], NLE requiresan additional sampling or inference step [14] to obtain the posterior distribution.
However, it allowsaccess to the intermediate likelihood approximation, a property we will later exploit to develop ourefficient feature selection algorithm
(see Sec. 2.3).
First, a set of N parameters are sampled from the prior distribution θ n ∼ p ( θ ) , n ∈ { 1 , . . .
, N } .
With these parameters, the simulator is run to implicitly sample from the model’s likelihood functionaccording to x n ∼ p ( x | θ n ) .
Here, x n = ( x 1 , . . .
, x N f ) are feature vectors that are usually takento be a function of the simulator output s with the individual features x i
=
f
i ( s ) , where i ∈{ 1 , . . . , N f } , rather than the output directly.
For mechanistic models whose output are time series, falso reduces the dimensionality of the data to a set of lower dimensional data features.
The resultingtraining data { x n , θ n } 1: N ∼ p ( x , θ ) can then be used to train a conditional density estimatorq ϕ
( x | θ ) , parameterized by ϕ , to approximate the likelihood function.
Here, we use a Mixture
Density Network for this task [29], the output density of which is parameterised by a mixture of
Gaussians, which can be marginalized analytically.
Thus, ˆ p ( x | θ ) =
(cid:80) k π k N ( µ k , Σ k ) with theparameters ( µ k , Σ k , π ) being non-linear functions of the inputs θ and the network parameters ϕ .
The parameters ϕ are optimized by maximizing the log-likelihood L ( x , θ )
= 1 N (cid:80) n log q ϕ ( x | θ )over the training data with respect to ϕ .
As the number of training samples goes to infinity, thisis equivalent to maximizing the negative Kullback-Leibler (KL) divergence between the true andapproximate posterior for every x ∈ supp ( x ) [28]:
[Equation]
After obtaining such a tractable likelihood surrogate, Bayes rule can be used to obtain an estimateof the posterior conditioned on an observation x o (see Eq. 2) for instance via Markov Chain Monte
Carlo (MCMC):
[Equation]
2.2 A naive algorithm for quantifying feature importance
Given this posterior estimate, we would now like to answer the following question: for which featurex i from a vector of features x does the uncertainty of the posterior estimate
ˆ p ( θ | x ) increase themost, when it is ignored?
For a single feature, a naive and costly algorithm does the following: iterateover x to obtain x \ i =
( x 1 , . . .
x i − 1 , x i +1 , . . .
, x N ) , train a total of N+1 density estimators to obtainthe likelihoods ˆ p ( x \ i | θ ) and ˆ p ( x | θ ) , sample their associated posteriors and compare ˆ p ( θ | x \ i ) to thereference posterior ˆ p ( θ | x ) with every feature present.
The same procedure can also be applied toquantify the contribution of any arbitrary subset of features.
As this procedure requires estimating thelikelihood based on the reduced feature set from scratch, for each feature (set) (Fig. 1, grey arrows),it is computationally costly.
To quantify the contribution of different features more efficiently, wewould thus need a way to avoid re-estimating the posterior with each feature left out.
- 5 θ5 θ5 θp( θ x ,x ,x ,x ) ∣ 0 1
Ground truth NLEFSLM θ o a5 θ5 θ5 θp( θ x ,x ,x ) ∣ 1 2 3 b5 θ5 θ5 θp( θ x ,x ,x ) ∣ 0 2 3 c5 θ5 θ5 θp( θ x ,x ,x ) ∣ 0 1 3 d5 θ5 θ5 θp( θ x ,x ,x ) ∣ 0 1 2 e θθθ
Ground truth x 0 x 1 x 2 xθθθ
FSLM
I I Q Q R R ( ( p p) )
̂
̂ \i
Removed feature
M ode l pa r a m e t e r s
Removed feature
M ode l pa r a m e t e r s f is clearly visible.
2.3 Efficient quantification of feature importance through post-hoc likelihood marginalization
To more efficiently estimate posteriors for any subset of features, we propose to marginalize thelikelihood estimate obtained via NLE post-hoc, as opposed to training a separate density estimatorfor every new set of features to obtain the marginal estimates (Sec. 2.2, Fig. 1).
We refer to thisalgorithm as efficient Feature Selection through Likelihood Marginalization (FSLM).
Suppose we want to evaluate the joint increase in posterior uncertainty for a subset of features.
Wepartition our feature vector x = [ x 1 , x 2 ] , such that x 2 contains the features to be removed from theinference.
To avoid the need for training the surrogate likelihood q ϕ for such a feature subset, we canrewrite the posterior with respect to x 1 as
[Equation]
Since we parameterize the approximate likelihood q ϕ as an MDN, we can perform this marginalizationanalytically: For a K-component MDN, the parameters of each mixture component can be similarlypartitioned to x [30, Chapter 2.3.2]:
[Equation]
Then the marginalization with respect to x 2 results in the following posterior distribution
[Equation]
[Table]
where π = π ( θ ) , µ = µ ( θ ) and Σ = Σ ( θ ) .
FSLM can thus sample from the posterior givenarbitrary feature subsets without estimating the surrogate likelihood q ϕ from scratch.
We implement FSLM using python 3.8, building on top of the public sbi library [31].
All the code isavailable at github.com/mnwncqvots/fslm.
All computations were done on an internal cluster running
Intel(R)
Xeon(R)
Gold 6226R CPUs @ 2.90GHz.
2.4 Measures of posterior uncertainty
Assuming the NLE procedure converges and the density estimator is sufficiently flexible, differencesin posterior uncertainty can be ascribed to differences in information content of features or noise in thedata.
Given that that noise in the data remains constant, relative differences in posterior uncertaintycan be used to assess the contributions of individual summary features.
Hence, we can identifyinformative summary statistics by quantifying the change in uncertainty of ˆ p ( θ | x \ i ) with respect toˆ p ( θ | x ) .
For this we use two metrics:
Ratio of IQR of the marginals: We use the ratio of inter quantile ranges (IQR) of the 1D-marginalsof ˆ p ( θ | x 1 ) and ˆ p ( θ | x ) .
This lets us deduce if specific features x i reduce the uncertainty of someparameters θ j more than others.
We use IQRs as a measure of spread as it is much less susceptible tooutliers compared to the standard deviation.
The resulting visualization allows to precisely pin downwhich parameter is affected by which summary feature.
Non-parametric estimate of KL divergence: As a second metric, we use a non-parametricestimate of the KL divergence between ˆ p ( θ | x 1 ) and ˆ p ( θ | x ) .
Since a posterior estimate, onaverage, cannot become more constrained if features are removed, increases in the KL estimateindicate an increase in uncertainty overall.
In order to estimate the KL-divergence between ˆ p ( θ | x 1 )and ˆ p ( θ | x ) , we employ a purely sampled based estimator [32, 33].
This is necessary because NLE only allows to evaluate unnormalized posteriors and hence circumvents having to evaluatethem explicitly.
The estimator makes use of 1-nearest neighbor search and is implemented usingk-d trees [34], to reduce the 2 N comparison operations to a manageable number.
With samples X =
{ X i } Ni =1 and Y =
{ Y i } Ni =1 from ˆ p ( θ | x 1 ) and ˆ p ( θ | x ) respectively, the KL-divergence KL (ˆ p ( θ | x 1 ) ||
ˆ p ( θ | x )) is estimated as
[Equation]
where d is the dimensionality of the samples, and || · || denotes the ℓ 2 -norm. [Figure] C g Na
[Figure] pa r a m [Figure] t e
[Figure] l [Figure] r a m
[Figure] t e r s a [Figure] p ( θ |x )
̂ \APA o e
Results
3.1 Example 1: Linear Gaussian model
We illustrate our algorithm in a simple toy problem, where by construction, we know which featuresshould have which effect.
We used a Linear Gaussian Model (LGM) with a three dimensionalparameter space θ =
[ θ 0 , θ 1 , θ 2 ] and four dimensional feature space x =
[ x 0 , x 1 , x 2 , x 3 ] .
θ isdrawn from a uniform prior θ i ∼ U ( − 5 , 5) , i ∈ { 0 , 1 , 2 } , with the features x being noisy lineartransformations of the parameter vector θ .
The mean µ ( θ ) and covariance Σ are defined byµ ( θ ) =
µ 0 + Lθ (7) Σ
= σ 2 1 .
With scale σ and shift µ 0 .
We construct L such that we can easily verify that the posterior reacts toand interacts with marginalized features as expected.
In our example, we let feature x 0 solely dependon θ 0 , x 1 only on θ 1 , x 2 on both θ 1 and θ 2 , and x 3 without influence on any model parameter. [Equation]
The resulting posterior is well concentrated on θ o and the NLE estimate matches the analyticallycomputed posterior well (Fig.
2 a ).
To show that FSLM can correctly identify individual featureimportance, while being more efficient than re-running NLE for each subset, we compared samplesfrom the FSLM-derived posterior estimates for every three feature subset to those obtained using theanalytical marginal likelihood p ( x \ i | θ ) , i ∈ { 0 , 1 , 2 , 3 } and the brute-force NLE estimate usingseparate q ϕ
( x \ i | θ ) for each subset (Fig. 2 b - e ).
We used a MDN with 10 mixture components andtrained on 10,000 samples.
Then 1,000 samples were drawn from each posterior, for a total of 10different initializations.
The posterior estimates obtained from FSLM matched those from the analytic ground truth veryaccurately (Fig.
2 b - e , and Tab. 1), while being several times faster to compute than NLE (see Tab.1).
As expected from the construction of our model, we observed that ˆ p ( θ | x \ 0 ) was not only lessconstrained compared to p ( θ | x ) (Fig.
2 a ), but that ˆ p ( θ 0 | x \ 0 )
=
p ( θ 0 )
=
U ( − 5 , 5) .
We also foundthat FSLM correctly identified the dependency structure between the other parameters and features(Fig.
2 c - e ), with FSLM matching the analytic ground truth while being much faster than NLE (Tab. 1).
3.2 Example 2: Hodgkin-Huxley neuron model
Next, we turned to a more realistic complex mechanistic model used for simulations of neurons.
Forour experiments we used a single compartment Hodgkin-Huxley model [3], Eq. 10, derived from
Pospischil et al.
In addition to the standard sodium, potassium and leak currents to generatespiking this model also considers a slow non-inactivating K + current to model spike-frequencyadaptation and a high-threshold Ca 2+ current to generate bursting.
[Equation]
Eq.
10 models the evolution of the membrane voltage V t
=
V ( t ) given a stimulus I ( t )
=
I t .¯
g i , i ∈
{ Na, K, l, M, L } are the maximum conductances of the sodium, potassium, leak, adaptivepotassium and calcium ion channels, respectively, and E i are the associated reversal potentials.
I tdenotes the current per unit area, C the membrane capacitance, A Soma the compartment area andn, m, h, q, r and p represent the fraction of independent gates in the open state, based on Hodgkinand Huxley [3].
For the remaining equations and parameter settings, see Appendix A.2.
Feature Explanation [Table] We then compared the performance of FSLM and bruteforce NLE to measure how commonly used summaryfeatures constrain the posterior of an exemplary voltagetrace.
Specifically, we used a set of 10 summary statistics commonly of interest to neurophysiologists, whichcapture the response dynamics of the HH model andcan be used to differentiate cortical cell types.
They arebriefly explained in Table 2.
To extract these featuresfrom the simulated traces we use the Allen SDK.
To train the density estimator we simulated 1 millionobservations with parameters drawn from a Uniformprior over biologically plausible parameter ranges (see A.2).
However, because a substantial fraction of thegenerated observations did not produce spiking behavior, some features such as spike rate were not defined.
Hence, these observations could not be used to trainthe density estimator.
We employed two different tech- a [Figure] 10 Run
Var(V ) m ¹(V ) m
APC latency
APA adapt
APW
APA
AHP
APT ¹(V ) rest
F ea t u r e 1 st 2 nd 3 rd 4 th 5 th 6 th 7 th 8 th 9 th 10 th
S e l e c t ed a s b niques introduced by Lueckmann et al.
[12] to reduce the amount of invalid samples returned by theprior and to compensate for any remaining invalid observations in the training data (for details see Appendix A.1).
This yielded accurate posterior estimates, despite incomplete data.
The change inuncertainty was then assessed on 3000 samples from ˆ p ( θ | x ) and ˆ p ( θ | x \ i ) .
[-Architectures and training were kept the same as in Sec.3.
-]
We found that both NLE and FSLM identified the same two features as yielding the highest increasein posterior uncertainty (Fig.
3 a ): the action potential threshold (APT) and the action potential count(APC).
Specifically, APT strongly affected the threshold voltage V T , the leak conductance g leak aswell as the sodium conductance g Na .
APC, on the other hand, influenced the leak reversal potential E leak , the leak conductance g leak , as well as C and g Na (Fig. 3 a ).
This result was independent ofwhether it was obtained by the brute-force algorithm or FSLM, however FSLM was substantiallyfaster ( ≈ 5 -fold speedup, Table1).
In contrast, the action potential amplitude (APA), as well as otherfeatures, did not have a sizable impact on the posterior in this example (Fig. 3 a ).
As a consequence, simulations based on samples from ˆ p ( θ | x ) and ˆ p ( θ | x \ APA ) were comparableto the observed voltage trace V o ( t ) , while the sample drawn from ˆ p ( θ | x \ APT ) showed differentspiking behavior, with much longer inter spike intervals in the simulation (Fig. 3 b ).
This resultedfrom higher values for the sodium conductance g Na and threshold voltage V T , compared to θ o , forthe particular simulation shown (Fig.
3 a , c ).
The measured increase in uncertainty is also directlyvisible in the two-dimensional marginal distributions of the posterior, where the removal of APTlead to a much less constrained posterior estimate (Fig.3 c ), while the removal of APA did not affectthe posterior uncertainty much (Fig.3 d ).
Also, the measured summary statistics from samples fromˆ p ( θ | x \ APT ) and ˆ p ( θ | x \ APA ) showed the expected behavior: the APT feature was much morevariable in samples from ˆ p ( θ | x \ APT ) than in ˆ p ( θ | x ) , while the estimated value for APA did notvary as drastically even for samples from ˆ p ( θ | x \ APA )
(Fig. 3 e ).
3.3 Efficient greedy feature selection using FSLM
In the previous section, we showed that FSLM can be used to study how the removal of individualfeatures increases the uncertainty of the posterior.
Next we show how FSLM can be used to studyhow the addition of features decreases uncertainty and provide a method to efficiently search forinformative features by greedy selection of new features.
First, we used NLE to obtain a posterior estimate given a candidate set of features.
We then used
FSLM to obtain all the 1D marginal likelihoods, sampled their associated posteriors
ˆ p ( θ | x i ) andcomputed their KL divergences to the full posterior.
We subsequently selected the feature x i thatminimizes the KL, before we repeated the procedure for all 2D marginals
ˆ p ( θ | ( x i , x j )) which include the selected feature x i .
We can iterate this procedure until a desired number of features isidentified.
To demonstrate this, we turned again to the HH example already studied in Sec. 3.2.
We used 1million samples for training and sampled every posterior 1000 times.
To check for consistency, werepeated the procedure 10 times.
Compared to selecting features at random, selecting features basedon the results of our greedy algorithm consistently yielded more constrained posterior estimates (see
Fig.
4 a ).
Furthermore, we found the three most informative features to be µ ( V rest ) , APT , AHP ,respectively, which were consistently chosen in this order (see Fig.
4 b ).
Discussion
We set out to develop an efficient method to characterize the relative contribution of different featuresto constraining the posterior in simulation-based inference.
To this end, we used NLE with a mixturedensity network to estimate the surrogate likelihood and marginalize it analytically.
We showedthat our method, FSLM, correctly identifies influential features both in a simple linear Gaussianmodel, as well as in a complex mechanistic model, namely the HH model in neuroscience.
In bothcases, FSLM, was as accurate as repeated application of NLE, but many times faster.
FSLM can thusprovide guidance on how to specify, prepare and select appropriate features or summary statistics, andalso allows one to study which parameters are constrained by which features to gain insight into thescientific structure of the model.
In contrast to methods developed for automatically deriving optimal(combinations of) summary statistics [16–19], FSLM focuses on making the evaluation and selectionof individual scientifically relevant feature sets interpretable.
Our visualization method allowed usto identify which features influenced the posterior uncertainty over which model parameters, and toidentify model parameters for which the posterior was not influenced by any feature.
For the case of a Hodgkin-Huxley model frequently used in neuroscience as a mechanistic modelof neural activity, we showed that action potential threshold and action potential count were themost informative features, jointly constraining the action potential threshold, the resting potential,membrane capacitance as well as model conductances (except for that of potassium).
Removingthese features from the inference procedure led to much more poorly constrained posteriors andsimulations that did not match the original trace.
We note some limitations of FSLM and the metrics introduced: First, the inherent dependence on
MDNs for the conditional neural density estimation limits overall accuracy for highly complexposteriors or likelihoods, as it only works well for likelihoods that are well approximated by mixturesof Gaussians.
Second, since our metrics are relative to the overall level of uncertainty in a givenposterior marginal, for very sharp distributions a small increase in variance can already lead to a largeincrease in our metrics, which would not be the case for a broad distribution.
While this may be thedesired behavior in some cases, it might be misleading in others.
Third, the effect of a feature may bejudged differently depending on whether the effect of removing it from a large set of features is beingassessed, or whether it is being added early in a greedy feature selection run.
Our method opens the door for careful studies on the relationship between data features and modeldistributions.
In neuroscience, for example, we could explore whether commonly used features suchas those used here constrain the posterior equally across different cellular families or types — forexample, is action potential threshold equally important for modeling fast-spiking interneurons andfor cellular families with more complex firing behavior, or are additional features needed in thiscase?
Given the efficient exploration of the relationship between features and posterior uncertainty,we believe that our method will also be applicable to a wide range of inference problems beyondneuroscience.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Sec. 4.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] We donot see negative social impact of our work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See Sec.2.3.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Sec.
3.1 and 3.2.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Sec. 2.3.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
[Table]
using/curating?
[Yes] No data from third parties was used.
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
[Table]
information or offensive content?
[Yes]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
[Table] (b) Did you describe any potential participant risks, with links to Institutional Review [Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A] [Table]