Efficient identification of informative features in simulation-based inference
Abstract
Simulation-based Bayesian inference (SBI) can be used to estimate the parameters of complex mechanistic models given observed model outputs without requiring access to explicit likelihood evaluations.
A prime example for the application of
SBI in neuroscience involves estimating the parameters governing the response dynamics of Hodgkin-Huxley (HH) models from electrophysiological measurements, by inferring a posterior over the parameters that is consistent with a set of observations.
To this end, many SBI methods employ a set of summary statistics or scientifically interpretable features to estimate a surrogate likelihood or posterior.
However, currently, there is no way to identify how much each summary statistic or feature contributes to reducing posterior uncertainty.
To address this challenge, one could simply compare the posteriors with and without a given feature included in the inference process.
However, for large or nested feature sets, this would necessitate repeatedly estimating the posterior, which is computationally expensive or even prohibitive.
Here, we provide a more efficient approach based on the SBI method neural likelihood estimation (NLE): We show that one can marginalize the trained surrogate likelihood post-hoc before inferring the posterior to assess the contribution of a feature.
We demonstrate the usefulness of our method by identifying the most important features for inferring parameters of an example HH neuron model.
Beyond neuroscience, our method is generally applicable to SBI workflows that rely on data features for inference used in other scientific fields.
Introduction
Mechanistic models are an elegant way to encode scientific knowledge about the world in the form of numerical simulations.
They include models such as Kepler’s laws of planetary motion [1], the SEIR model [2] for describing the spread of infectious diseases or the Hodgkin-Huxley (HH) model for the dynamics of action potentials in neurons [3].
Efficiently constraining the parameters of such models by measurements is a key problem in many disciplines [4–7].
Since these models give rise to intractable likelihood functions, however, classical likelihood-based Bayesian methods such as variational inference [8] or Markov Chain Monte Carlo [9] cannot be used directly.
There are algorithms, p r obab ili t y feature(marginal) likelihood f ea t u r efeaturep r obab ili t y featureposterior pa r a m e t e r parameterpa r a m e t e r parameterpa r a m e t e r parameter
MDN prior p r obab ili t y pa r a m e t e r parametersimulated data
[Figure]
such as ABC-MCMC [10] or pseudo-marginal methods [11] that deal with this problem, however, they have potentially slow convergence rates and are computationally expensive.
To overcome this problem, several techniques collectively known as simulation-based inference (SBI) have recently been developed [12].
Leveraging the ability to simulate the model, these techniques obtain estimates of the likelihood or posterior from simulated data.
The most recent of these algorithms such as neural likelihood estimation (NLE) [13] or neural posterior estimation (NPE) [14–16] employ state-of-the-art neural density estimators to learn tractable surrogates of these functions to be evaluated instead of the real quantities.
To this end, summary statistics or features capturing essential aspects of the model’s dynamics are defined by the scientist to reduce the high dimensional model output to manageable scale, in turn decreasing the problem complexity and computation costs.
This is often done by hand to emphasize specific aspects of the data or to aid scientific interpretation [12], but can also be automated [17–23].
In neuroscience, many different approaches have been developed in order to find suitable parameters of models of neural activity [4, 5, 24–29].
SBI approaches have been used to infer parameters in biophysical neuron models from measurements of neural activity in a Bayesian way [15, 30, 31].
For inference of HH models from electrophysiological data, features such as action potential threshold or width and resting membrane potential or spike count can be used, reflecting measures that electrophysiologists use to characterize recorded neurons.
For scientific interpretation of SBI results, it is often of interest which features, or combinations of features, have the biggest impact on the posterior and which parameters they affect specifically.
To this end, one could compare the posterior uncertainty estimated with and without including a specific feature in the SBI method — the increase in uncertainty resulting from not relying on that feature can be used as a measure of its importance and on average is equivalent to its mutual information.
Of course, this approach can also be applied to whole subsets of features.
To evaluate an entire set of features exhaustively would require re-estimating the posterior many times (Fig. 1, grey), which would scale prohibitively with the number of features.
To address this issue, we here introduce a method called Feature Selection Through Likelihood Marginalization (FSLM) to compute posteriors for arbitrary subsets of features, without the need for repeated training (Fig. 1, black).
To achieve this, we here use NLE [32] and exploit the marignalization properties of mixture density networks
(MDNs) [33], which can be used with NLE as a density estimator: instead of re-estimating the surrogate likelihood from scratch, we marginalize it analytically with respect to a given (set of) features before applying Bayes rule to obtain the posterior estimate.
This way, we can efficiently compare the posterior uncertainty with and without including a certain feature.
For a simple linear Gaussian model and non-linear HH models, we show that the obtained posterior estimates are as accurate and robust as when repeatedly retraining density estimators from scratch, despite being much faster to compute.
We then apply our algorithm to study which features are the most useful for constraining posteriors over parameters of the HH model.
Our tool allows us also to study which model parameter is affected by which summary feature.
Finally, we suggest a greedy feature selection strategy to select useful features from a predefined set.
Methods 2.1 Neural Likelihood Estimation (NLE)
Neural Likelihood Estimation [32] is a SBI method which approximates the likelihood p ( x | θ ) of the data x given the model parameters θ by training a conditional neural density estimator on data generated from simulations of a mechanistic model.
Using Bayes rule, the approximate likelihood can then be used to obtain an estimate of the posterior (Fig. 1).
Unlike NPE [15, 16], NLE requires an additional sampling or inference step [34] to obtain the posterior distribution.
However, it allows access to the intermediate likelihood approximation, a property we will later exploit to develop our efficient feature selection algorithm
(see Sec. 2.3).
First, a set of N parameters are sampled from the prior distribution θ n ∼ p ( θ ) , n ∈ { 1 , . . .
, N } .
With these parameters, the simulator is run to implicitly sample from the model’s likelihood function according to x n ∼ p ( x | θ n ) .
Here, x n = ( x 1 , . . .
, x N f ) are feature vectors that are usually taken to be a function of the simulator output s with the individual features x i
=
f
i ( s ) , where i ∈ { 1 , . . . , N f } , rather than the output directly.
For mechanistic models whose output are time series, f also reduces the dimensionality of the data to a set of lower dimensional data features.
The resulting training data { x n , θ n } 1: N ∼ p ( x , θ ) can then be used to train a conditional density estimator q ϕ ( x | θ ) , parameterized by ϕ , to approximate the likelihood function.
Here, we use a Mixture
Density Network for this task [33], the output density of which is parameterised by a mixture of
Gaussians, which can be marginalized analytically.
Thus, ˆ p ( x | θ ) =
(cid:80) k π k N ( µ k , Σ k ) with the parameters ( µ k , Σ k , π ) being non-linear functions of the inputs θ and the network parameters ϕ .
The parameters ϕ are optimized by maximizing the log-likelihood L ( x , θ )
= 1 N (cid:80) n log q ϕ ( x | θ ) over the training data with respect to ϕ .
As the number of training samples goes to infinity, this is equivalent to maximizing the negative Kullback-Leibler (KL) divergence between the true and approximate posterior for every x ∈ supp ( x ) [32]:
[Equation]
After obtaining such a tractable likelihood surrogate, Bayes rule can be used to obtain an estimate of the posterior conditioned on an observation x o (see Eq. 2) for instance via Markov Chain Monte
Carlo (MCMC):
[Equation]
2.2 A naive algorithm for quantifying feature importance
Given this posterior estimate, we would now like to answer the following question: for which feature x i from a vector of features x does the uncertainty of the posterior estimate
ˆ p ( θ | x ) increase the most, when it is ignored?
For a single feature, a naive and costly algorithm does the following: iterate over x to obtain x \ i =
( x 1 , . . .
x i − 1 , x i +1 , . . .
, x N ) , train a total of N+1 density estimators to obtain the likelihoods ˆ p ( x \ i | θ ) and ˆ p ( x | θ ) , sample their associated posteriors and compare ˆ p ( θ | x \ i ) to the reference posterior ˆ p ( θ | x ) with every feature present.
The same procedure can also be applied to quantify the contribution of any arbitrary subset of features.
As this procedure requires estimating the likelihood based on the reduced feature set from scratch for each feature (set) (Fig. 1, grey arrows), it is computationally costly.
To quantify the contribution of different features more efficiently, we would thus need a way to avoid re-estimating the posterior with each feature left out.
- 5 θ5 θ5 θp( θ x ,x ,x ,x ) ∣ 0 1
Ground truth NLE+MDNFSLM θ o a5 θ5 θ5 θp( θ x ,x ,x ) ∣
1 2 3 b θθθ
Ground truth x 0 x 1 x 2 xθθθ
FSLM
I I Q Q R R ( ( p p) )
̂
̂ \i
Removed feature
M ode l pa r a m e t e r s
Removed feature
M ode l pa r a m e t e r s c
2.3 Efficient quantification of feature importance through post-hoc likelihood marginalization
To more efficiently estimate posteriors for any subset of features, we propose to marginalize the likelihood estimate obtained via NLE post-hoc, as opposed to training a separate density estimator for every new set of features to obtain the marginal estimates (Sec. 2.2, Fig. 1).
We refer to this algorithm as efficient Feature Selection through Likelihood Marginalization (FSLM).
Suppose we want to evaluate the joint increase in posterior uncertainty for a subset of features.
We partition our feature vector x = [ x 1 , x 2 ] , such that x 2 contains the features to be removed from the inference.
To avoid the need for training the surrogate likelihood q ϕ for such a feature subset, we can rewrite the posterior with respect to x 1 as
[Equation]
Since we parameterize the approximate likelihood q ϕ as an MDN, we can perform this marginalization analytically:
For a K-component MDN, the parameters of each mixture component can be similarly partitioned to x [35, Chapter 2.3.2]:
[Equation]
Then the marginalization with respect to x 2 results in the following posterior distribution
[Equation]
where π = π ( θ ) , µ = µ ( θ ) and Σ = Σ ( θ ) .
FSLM can thus sample from the posterior given arbitrary feature subsets without estimating the surrogate likelihood q ϕ from scratch.
We implement FSLM using python 3.8, building on top of the public sbi library [36].
All the code is available at github.com/berenslab/fslm_repo.
All computations were done on an internal cluster running Intel(R)
Xeon(R)
Gold 6226R CPUs @ 2.90GHz.
2.4 Measures of posterior uncertainty
Assuming the NLE procedure converges and the density estimator is sufficiently flexible, differences in posterior uncertainty can be ascribed to differences in information content of features or noise in the data.
Given that that noise in the data remains constant, relative differences in posterior uncertainty can be used to assess the contributions of individual summary features.
Hence, we can identify informative summary statistics by quantifying the change in uncertainty of ˆ p ( θ | x \ i ) with respect to ˆ p ( θ | x ) . For this we use two metrics:
[Table]
Ratio of IQR of the marginals: We use the ratio of inter quantile ranges (IQR) of the 1D-marginals of ˆ p ( θ | x 1 ) and ˆ p ( θ | x ) .
This lets us deduce if specific features x i reduce the uncertainty of some parameters θ j more than others.
We use IQRs as a measure of spread as it is much less susceptible to outliers compared to the standard deviation.
The resulting visualization allows to precisely pin down which parameter is affected by which summary feature.
Non-parametric estimate of KL divergence: As a second metric, we use a non-parametric estimate of the KL divergence between ˆ p ( θ | x 1 ) and ˆ p ( θ | x ) .
Since a posterior estimate, on average, cannot become more constrained if features are removed, increases in the KL estimate indicate an increase in uncertainty overall.
To estimate the KL-divergence between ˆ p ( θ | x 1 ) and ˆ p ( θ | x ) , we employ a purely sample based estimator [37, 38].
This is necessary because NLE only allows to evaluate unnormalized posteriors which circumvents having to evaluate them explicitly.
The estimator makes use of 1-nearest neighbor search and is implemented using k-d trees [39], to reduce the 2 N comparison operations to a manageable number.
With samples X
=
{ X i } Ni =1 and
Y =
{ Y i } Mi =1 from ˆ p ( θ | x 1 ) and ˆ p ( θ | x ) respectively, the KL-divergence KL (ˆ p ( θ | x 1 )
||
ˆ p ( θ | x )) is estimated as
[Equation]
where d is the dimensionality of the samples, and || · || denotes the ℓ 2 -norm.
We note that the Maximum Mean Discrepancy (MMD) [40, 41] could be used as an alternative measure, but preliminary experiments indicated that the KL estimate was more consistent.
Results 3.1 Example 1: Linear Gaussian model
We illustrate our algorithm in a simple toy problem, where by construction, we know which features should have which effect.
We used a Linear Gaussian Model (LGM) with a three dimensional parameter space θ =
[ θ 0 , θ 1 , θ 2 ] and four dimensional feature space x =
[ x 0 , x 1 , x 2 , x 3 ] .
θ is drawn from a uniform prior θ i ∼ U ( − 5 , 5) , i ∈ { 0 , 1 , 2 } , with the features x being noisy linear transformations of the parameter vector θ .
The mean µ ( θ ) and covariance Σ are defined by
[Equation]
With scale σ and shift µ 0 .
We construct L such that we can easily verify that the posterior reacts to and interacts with marginalized features as expected.
In our example, we let feature x 0 solely depend on θ 0 , x 1 only on θ 1 , x 2 on both θ 1 and θ 2 , and x 3 without influence on any model parameter.
[Figure]
AP W A H P AP
[Figure]
APA adap
t l a t en cy μ ( V ) r e
s t μ ( V
[Figure]
m
[Figure]
a r(
[Figure] [Figure]
C g Na g K g M g leak g L τ max V T
E leak r SS
NLE +
[Figure]
AP
[Figure]
AP
[Figure]
A H
[Figure]
AP
[Figure]
APA adap
[Figure]
t
[Figure]
μ ( V
[Figure]
μ
[Figure] [Figure]
μ ( V
[Figure] [Figure]
[Figure] [Figure]
M ode l pa r a m e t e r s
[Figure]
ode l pa r a m e t e r s
[Figure]
ode l pa
[Figure]
a m e t
e
r s
[Figure]
o f po s t e r i
o r s a m p l e s ( m V ) x
o p( θ |x )
̂ o p ( θ |x )
̂ \APT o p ( θ |x )
̂ \APA o e
The resulting posterior is well concentrated on θ o and the NLE+MDN estimate matches the analytically computed posterior well (Fig.
2 a ).
To show that FSLM can correctly identify individual feature importance, while being more efficient than re-running NLE for each subset, we compared samples from the FSLM-derived posterior estimates for every three feature subset to those obtained using the analytical marginal likelihood p ( x \ i | θ ) , i ∈ { 0 , 1 , 2 , 3 } and brute-force NLE estimates using separate q ϕ ( x \ i | θ ) for each subset (Fig. 2 b , Appendix Fig.5 a - c ).
The brute-force baseline was computed with both a MDN and a Masked Autoregressive Flow (MAF) [42].
The MDNs had three hidden layers and 10 mixture components.
The MAF consisted of five 3 layer deep MADEs [43].
Training was done on 10,000 samples, then 500 samples were drawn from each posterior, for a total of 10 different initializations.
Training was terminated when the log-likelihoods had stopped improving on a validation set that was split from the training set at 10% over 20 epochs.
The posterior estimates obtained from FSLM matched those from the analytic ground truth very accurately (Fig.
2 b - e , Appendix Fig.
5 a - c and Tab. 1), while being several times faster to compute than both NLE alternatives (see Tab. 1).
As expected from the construction of our model, we observed that ˆ p ( θ | x \ 0 ) was not only less constrained compared to p ( θ | x ) (Fig.
2 a ), but that ˆ p ( θ 0 | x \ 0 )
=
p ( θ 0 )
=
U ( − 5 , 5) .
We also found that FSLM correctly identified the dependency structure between the other parameters and features (Appendix Fig.
5 a - c ), with FSLM matching the analytic ground truth while being much faster than both NLE approaches (Tab. 1).
3.2 Example 2: Hodgkin-Huxley neuron model
Next, we turned to a more realistic complex mechanistic model used for simulations of neurons.
For our experiments we used a single compartment Hodgkin-Huxley model [3], Eq. 8, derived from
Pospischil et al.
In addition to the standard sodium, potassium and leak currents to generate spiking this model also considers a slow non-inactivating K + current to model spike-frequency adaptation and a high-threshold Ca 2+ current to generate bursting.
[Equation]
Eq.
8 models the evolution of the membrane voltage V t
=
V ( t ) given a stimulus I ( t )
=
I t .
¯ g i , i ∈
Na, K, l, M, L } are the maximum conductances of the sodium, potassium, leak, adaptive potassium and calcium ion channels, respectively, and E i are the associated reversal potentials.
I t denotes the current per unit area, C the membrane capacitance, A Soma the compartment area and n, m, h, q, r and p represent the fraction of independent gates in the open state, based on Hodgkin and Huxley [3].
For the remaining equations and parameter settings, see Appendix A.3.
We then compared the performance of FSLM and the two versions of brute-force NLE to measure how commonly used summary features constrain the posterior of an exemplary voltage trace.
Specifically, we used a set of 10 summary statistics commonly of interest to neurophysiologists, which capture the response dynamics of the HH model and can be used to differentiate cortical cell types.
They are briefly explained in Table 3.
To extract these features from the simulated traces we use the Allen
SDK.
To train the density estimator we simulated 1 million observations with parameters drawn from a
Uniform prior over biologically plausible parameter ranges (see A.3).
However, because a substantial fraction of the generated observations did not produce spiking behavior, some features such as spike rate were not defined.
Hence, these observations could not be used to train the density estimator.
We employed two different techniques introduced by Lueckmann et al.
[15] to reduce the amount of invalid samples returned by the prior and to compensate for any remaining invalid observations in the training data (for details see Appendix A.2).
This yielded accurate posterior estimates, despite incomplete data.
The change in uncertainty was then assessed on 3000 samples from the MDN estimates of ˆ p ( θ | x ) and ˆ p ( θ | x \ i ) .
Architectures and training were kept the same as in Sec.3.
We found that all three algorithms identified the same two features as yielding the highest increase in posterior uncertainty (Fig.
3 a ): the action potential threshold (APT) and the action potential count (APC), while a MAF additionally identified the mean resting potential ( µ ( V rest ) ) as a constraining feature.
For the MDN-based density estimates, APT strongly affected the threshold voltage V T , the a
[Figure]
Var(V ) m μ (
[Figure]
[Figure]
APC latency
APA adapt
APW
APA
AHP
APT μ (V ) rest
F ea t u r e 1 st 2 nd 3 rd 4 th 5 th 6 th 7 th 8 th 9 th 10 th
S e l e c t ed a s b leak conductance g leak as well as the sodium conductance g Na .
APC, influenced the leak reversal potential E leak , the leak conductance g leak , as well as C and g Na (Fig.
3 a ).
When a flow was used, g NA did not seem to be constrained by either APT or APC and µ ( V rest ) additionally constrained the leak reversal potential ( E leak ), as well as the adaptive Potassium current g M .
In contrast, the action potential amplitude (APA), as well as other features, did not have a sizable impact on the posterior in this example (Fig. 3 a ).
While the results for both MDN-based algorithms are equivalent and have strong overlap with the flow-based predictions, FSLM was significantly faster than both brute-force alternatives ( ≈ 5 -fold and ≈ 7 -fold speedups respectively, Table 1) and overall almost as accurate, when measured against a common baseline (Table 1).
As a consequence, simulations based on samples from ˆ p ( θ | x ) and ˆ p ( θ | x \ APA ) were comparable to the observed voltage trace V o ( t ) , while the sample drawn from ˆ p ( θ | x \ APT ) showed different spiking behavior, with much longer inter spike intervals in the simulation (Fig. 3 b ).
This resulted from higher values for the sodium conductance g Na and threshold voltage V T , compared to θ o , for the particular simulation shown (Fig.
3 a , c ).
The measured increase in uncertainty is also directly visible in the two-dimensional marginal distributions of the posterior, where the removal of APT lead to a much less constrained posterior estimate (Fig.3 c ), while the removal of APA did not affect the posterior uncertainty much (Fig.3 d ).
Also, the measured summary statistics from samples from ˆ p ( θ | x \ APT ) and ˆ p ( θ | x \ APA ) showed the expected behavior: the APT feature was much more variable in samples from ˆ p ( θ | x \ APT ) than in ˆ p ( θ | x ) , while the estimated value for APA did not vary as drastically even for samples from ˆ p ( θ | x \ APA )
(Fig. 3 e ).
3.3 Efficient greedy feature selection using FSLM
In the previous section, we showed that FSLM can be used to study how the removal of individual features increases the uncertainty of the posterior.
Next we show how FSLM can be used to study how the addition of features decreases uncertainty and provide a method to efficiently search for informative features by greedy selection of new features.
First, we used NLE to obtain a posterior estimate given a candidate set of features.
We then used
FSLM to obtain all the 1D marginal likelihoods, sampled their associated posteriors
ˆ p ( θ | x i ) and computed their KL divergences to the full posterior.
We subsequently selected the feature x
i that minimizes the KL, before we repeated the procedure for all 2D marginals ˆ p ( θ | ( x i , x j )) which include the selected feature x i .
We can iterate this procedure until a desired number of features is identified.
This is equivalent to a beam-search [45] with a beam size equal to one.
To investigate combinations of features the scheme is easily extensible to larger beam sizes.
To demonstrate this, we turned again to the HH example already studied in Sec. 3.2.
We usedmillion samples for training and sampled every posterior 1000 times.
To check for consistency, we repeated the procedure 50 times.
Compared to selecting features at random, selecting features based on the results of our greedy algorithm consistently yielded more constrained posterior estimates (see
Fig.
4 a ).
Furthermore, we found the three most informative features to be µ ( V rest ) , APT , AHP , respectively, which were consistently chosen in this order (see Fig.
4 b ).
Discussion
We set out to develop an efficient method to characterize the relative contribution of different features to constraining the posterior in simulation-based inference.
To this end, we used NLE with a mixture density network to estimate the surrogate likelihood and marginalize it analytically.
We showed that our method, FSLM, correctly identifies influential features both in a simple linear Gaussian model, as well as in a complex mechanistic model, namely the HH model in neuroscience.
In both cases FSLM was as accurate as repeated application of NLE, but many times faster, independent of whether the likelihood was approximated by MDNs or MAFs.
While the different architectures lead to subtle differences in their predictions of important features of HH models, we found them still to be mostly in agreement.
FSLM can thus provide guidance on how to specify, prepare and select appropriate summary statistics, while also allowing to study which parameters are constrained by which features to gain insight into the scientific structure of the model.
In contrast to methods developed for automatically deriving optimal (combinations of) summary features [17–19, 21, 23],
FSLM focuses on making the evaluation and selection of individual scientifically relevant feature sets interpretable.
Our visualization method allowed us to identify which features influenced the posterior uncertainty over which model parameters, and for which model parameters the posterior was not influenced by any feature.
A similar approach could also be implemented for Approximate Bayesian
Computation (ABC) [46, 47].
Compared to approaches that aim to directly quantify the mutual information (MI) between sets of parameters and features [23], our method aims to quantify the relationship between model parameters and features for individual observations.
Computing estimates of the MI for simulator models is difficult without access to a closed form likelihood and purely sample based estimators [48, 49] do not allow for inspection of individual or small sets of observations.
However, this is of interest when different cells, cell types or cell families are constrained by different feature sets.
Nonetheless, FSLM still retains the option to obtain a single ranking of features across a whole dataset by averaging over all observations in the set, which would then be equivalent to the MI between posterior and prior (the trained neural network is amortized and, thus, requires no retraining for different observations).
For the case of a Hodgkin-Huxley model, frequently used in neuroscience as a mechanistic model of neural activity, we showed that action potential threshold and action potential count were the most informative features, jointly constraining the action potential threshold, the resting potential, membrane capacitance as well as model conductances (except for that of potassium).
Removing these features from the inference procedure led to much more poorly constrained posteriors and simulations that did not match the original trace.
We note some limitations of FSLM and the metrics introduced: First, the inherent dependence on
MDNs for the conditional neural density estimation limits overall accuracy for highly complex posteriors or likelihoods, compared to e.g. normalizing flows, as it only works well for likelihoods that are well approximated by mixtures of Gaussians.
Additionally, the use of full covariance matrices can be problematic in higher-dimensional feature spaces due to the curse of dimensionality and one would have to evaluate whether the use of e.g. diagonal covariances is more suitable.
Second, since our metrics are relative to the overall level of uncertainty in a given posterior marginal, for very sharp distributions a small increase in variance can already lead to a large increase in our metrics, which would not be the case for a broad distribution.
While this may be the desired behavior in some cases, it might be misleading in others.
Third, the effect of a feature may be judged differently depending on whether the effect of removing it from a large set of features is being assessed, or whether it is being added early in a greedy feature selection run.
Finally, the approximation quality of NLE, thus also FSLM, depends crucially on the choice of prior [50].
However, posterior predictive checks [51, 52] and coverage tests [53] applicable to NLE can also be applied to FLSM to verify the correctness of the inference results.
Our method opens the door for careful studies on the relationship between data features and model distributions.
In neuroscience, for example, we could explore whether commonly used features such as those used here constrain the posterior equally across different cellular families or types — for example, is action potential threshold equally important for modeling fast-spiking interneurons and for cellular families with more complex firing behavior, or are additional features needed in this case?
Given the efficient exploration of the relationship between features and posterior uncertainty, we believe that our method will also be applicable to a wide range of inference problems beyond neuroscience.
Acknowledgments
We thank the German Research Foundation for support through the Excellence Cluster "Machine
Learning — New Perspectives for Science" (EXC 2064, 390727645) and a Heisenberg Professorship to PB (BE5601/8-1).
JHM and PB are also members of the Tübingen AI Center funded by the
German Ministry for Science and Education (FKZ 01IS18039A).
We thank the International Max
Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Michael Deistler.
We also thank our colleagues, in particular Jonathan Oesterle and Lisa Koch, for their valuable comments and fruitful discussions.
A.2 Excluding invalid simulations
The simulator for the HH model can produce voltage dynamics that do not exhibit any spiking.
This will lead to undefined values (NaN) in the spike related summary features that we use (see Tab. 3).
In direct posterior estimation methods [15, 16], the invalid training pairs can simply be ignored and dropped from the dataset.
However, this is not the case for methods learning the likelihood, as this leads to likelihood estimates that are unaware of parameter regions which produce invalid simulations and hence a biased estimate according to
[Equation]
(9) with ℓ ϕ being the biased likelihood estimate.
To compensate for this mismatch we therefore employ two techniques also employed by Lueckmann et al.
[15] and Glöckler, Deistler, and Macke [34] (for proofs see Glöckler, Deistler, and Macke [34] A.6; Theorem 1, Lemma 1 and Lemma 3).
First, we restrict the prior to reduce the fraction of invalid training examples that are produced in the first place, by training a probabilistic classifier to only accept parameter samples that are predicted to result in valid observations.
The classifier we use is just a simple fully connected neural net with a softmax output, which is trained with a negative log-likelihood loss.
Hence the restricted prior comes out to be
[Equation]
, where p ( y ) is the probability of θ producing valid observations and p ( y | θ ) is the binary probabilistic classifier that predicts whether a set of parameters θ is likely to produce observations that include non valid (NaN or inf) features.
We only expend a small amount of simulation budget to do this.
Since, a small fraction of samples still contains non-sensical observations which have to be withheld from training, we still have to adjust for the remaining bias of the likelihood ℓ ψ .
This is achieved by estimating c ζ ( θ )
= p ( y = valid | θ ) on the entirety of the training data, such that:
[Equation]
We implement c ζ ( θ ) as a simple logistic regressor that predicts whether a set of parameters θ produces valid observations features.
Unlike Glöckler, Deistler, and Macke [34] however, we opt to combine this calibration term with the prior, rather than the likelihood.
This is mathematically equivalent according to Eq.
and is easier to implement in practice.
A.3 The Hodgkin Huxley Model
The gating dynamics of the HH model are expressed through Eq.
12, 13, where α z ( V t ) and β z ( V t ) denote the rate constants for one of the gating variables z ∈ { m, n, h, q, r } .
They model the different voltage dependencies for each channel type and their parameterisation depends on the specific model that is used.
In this work we stick to the kinetics of α z ( V t , V T ) , β z ( V t , V T ) , p ∞ ( V t ) and τ p ( V t , τ max ) as modeled in Pospischil et al.
[Equation]
In our experiments we simulate response of the membrane voltage to the injection of a square wave of depolarizing current of 200 pA .
Since the system of equations is not solvable analytically, we compute the membrane voltage V t
=
V ( t ) iterating over the time axis in steps of dt = 0 .
04 ms using the exponential Euler method [54] as it provides a good trade-off between computational complexity and accuracy.
The initial voltage is chosen as V 0 =
70 mV the membrane voltage V ( t ) is simulated for a time interval of 800 ms .
For the parameters we used the following values or prior ranges:
Parameter
Lower bound
Upper bound
C 0.4 µF/cm
µF/cmg Na 0.5 mS 80 mS g K 1 · 10 − 4 mS 30 mS g M − 3 · 10 − 5 mS 0.6 mS g leak 1 · 10 − 4 mS 0.8 mS g L − 3 · 10 − 5 mS 0.6 mS τ max 50 s 3000 s
V T -90 mV -40 mV
E leak -110 mV -50 mV r SS 0.
Parameter
[Table]
where A Soma = τ CR in , C the membrane capacitance, R in the input resistance.
Furthermore, τ is the membrane time constant, τ max the time constant of the slow K + current and V T the threshold voltage.
We also include two additional constants, k
[Equation]
and r SS , which adjust for the ambient temperature and for different rate scaling respectively.
The output of the HH model is summarized in terms of summary features, which are explained in
Tab. 3.
A.4 The full posterior estimates for the HH model
Here we provide the 1D and 2D marginal plots for all parameters of the posterior estimates ˆ p ( θ | x ) , ˆ p ( θ | x \ APT ) and ˆ p ( θ | x \ APA ) in Fig.3 c,d .
[Figure] [Table]