Towards Reliable Simulation-Based Inference with Balanced Neural Ratio Estimation
Abstract
Modern approaches for simulation-based inference rely upon deep learning surrogates to enable approximate inference with computer simulators.
In practice, the estimated posteriors’ computational faithfulness is, however, rarely guaranteed.
For example, Hermans et al.
[1] show that current simulation-based inference algorithms can produce posteriors that are overconﬁdent, hence risking false inferences.
In this work, we introduce Balanced Neural Ratio Estimation ( BNRE ), a variation of the NRE algorithm
[2] designed to produce posterior approximations that tend to be more conservative, hence improving their reliability, while sharing the same Bayes optimal solution.
We achieve this by enforcing a balancing condition that increases the quantiﬁed uncertainty in small simulation budget regimes while still converging to the exact posterior as the budget increases.
We provide theoretical arguments showing that BNRE tends to produce posterior surrogates that are more conservative than NRE ’s. We evaluate BNRE on a wide variety of tasks and show that it produces conservative posterior surrogates on all tested benchmarks and simulation budgets.
Finally, we emphasize that BNRE is straightforward to implement over NRE and does not introduce any computational overhead.
Introduction
Many areas of science and engineering use parametric computer simulations to describe complex stochastic generative processes.
In this setting, Bayesian inference provides a principled framework to identify parameters matching empirical observations.
Computer simulations, however, deﬁne the necessary likelihood function only implicitly, which prevents its evaluation and the use of classical inference algorithms.
To overcome this obstacle, recent simulation-based inference (SBI) algorithms [3] build upon deep learning surrogates to approximate parts of the Bayes rule and enable approximate inference.
For example, [4, 5] build a surrogate of the likelihood function while [6, 7, 2, 8, 9] approximate the likelihood-to-evidence ratio.
The posterior can also be targeted directly with variational inference, as proposed by [10, 11, 5].
These algorithms are either amortized or run sequentially to drive the training towards a target observation and improve the simulation efﬁciency of the procedure [10, 12, 2, 11, 4, 8, 5].
However, sequential methods have the drawback of being computationally expensive to diagnose as the surrogates are only valid for the target observation [1].
Truncated marginal neural ratio estimation [9] alleviates this issue by introducing a sequential algorithm that builds a surrogate valid in a local region around the target.
Since modern simulation-based inference algorithms rely on deep learning surrogates, concerns naturally arise regarding their computational faithfulness and whether they are sufﬁciently adequate for the inference task of interest.
In Bayesian inference, these concerns can be at least partially addressed with diagnostics designed to probe the correct behaviour of the inference method, such as ˆ
R diagnostics for MCMC [13], or to assess the quality of posterior approximations directly.
The latter include diagnostics such as simulation-based calibration (SBC) [14] or coverage-based diagnostics [15, 1].
As discussed by Hermans et al.
[1], posterior approximations must be conservative to guarantee reliable inferences, even when approximations are not faithful.
For example, in the physical sciences, where the goal is often to constrain parameters of interest, wrongly excluding plausible values could drive the scientiﬁc inquiry in the wrong direction, whereas failing to exclude implausible values because of (too) conservative estimations is much less detrimental.
Unfortunately, the same authors also demonstrate that current simulation-based inference algorithms can lead to overconﬁdent surrogates and therefore false inferences.
Scientiﬁc use cases requiring conservative inference include for example the study of dark matter models in particle physics and astrophysics [16], which could be cold, warm, or hot dark matter.
In general, thermal dark matter models are described by a single parameter, the dark matter thermal relic mass, which can be intuitively thought of as the energy the dark matter particle had in the Early
Universe.
Small values correspond to warm or hot dark matter, while high values are descriptive of cold dark matter.
Applying an inference algorithm without diagnosing the learned estimator could lead to constraints that are tighter than they should be.
For example, whenever an overconﬁdent estimator produces posterior estimates that favor cold dark matter models, it could simultaneously reject alternative models, such as the extensively studied Sterile Neutrino, a potential candidate for the Warm Dark Matter particle.
Making a scientiﬁc statement in this direction therefore requires the uttermost care to not wrongly exclude values of the thermal relic mass that are actually plausible.
In this work, we develop a novel algorithm that not only converges to exact inference as the simulation budget increases, but which is also more likely to produce conservative surrogates in small simulation budget regimes.
Towards this objective, we propose a variant of the NRE algorithm called Balanced
Neural Ratio Estimation ( BNRE ), which enforces a balancing condition on the binary neural classiﬁer to increase the reliability of its posterior approximations.
The structure of the manuscript is outlined as follows.
Section 2 describes the formalism and the necessary background.
Section 3 describes BNRE and provides theoretical arguments towards its conservativeness and reliability.
Section 4 illustrates our main results and provides insights regarding the behaviour of the method.
Finally, Section 5 discusses related work while Section 6 summarizes our contributions and hints at future work.
Background 2.1 Statistical formalism
This work is concerned with simulation-based inference algorithms that produce posterior approximations ˆ p ( ϑ | x ) under the following semantics.
Target parameters ϑ denote the parameters of the model and we make the reasonable assumption that the prior p ( ϑ ) is tractable.
The model is generically expressed as a computer program, a simulator, that describes the forward dynamics of interest based on the input parameters ϑ .
The simulator implicitly deﬁnes the likelihood function p ( x | ϑ ) .
While we cannot directly evaluate the likelihood p ( x | ϑ ) , we can execute the computer program to generate synthetic observables x ∼ p ( x | ϑ ) .
Every observable x o is tied to ground truth parameters
ϑ ∗ whose forward evaluation within the simulator produced x ∗ .
Of special importance to Bayesians is the notion of a credible region , which is a domain Θ within the target parameter space that satisﬁes
[Equation]
for some
[Equation]
conﬁdence level 1 − α .
Because many such regions exist, we target the credible region with the smallest volume, also known as the highest posterior density region [17, 18].
2.2 Neural ratio estimation
Neural Ratio Estimation ( NRE ) is an established approach in the simulation-based inference literature both from frequentist [6] and Bayesian [7, 2, 8, 9] perspectives.
In essence, all protocols rely on the density-ratio trick [19, 20, 6] to construct a surrogate of the likelihood ratio.
In this work, we consider an amortized estimator
ˆ r ( x | ϑ ) of the intractable likelihood-to-evidence ratio r ( x | ϑ ) =
p ( ϑ , x ) /p
( ϑ ) p ( x ) =
p ( x | ϑ ) /p
( x ) that can be learned by training a binary classiﬁer ˆ d : X × Θ (cid:55)→ [0 , 1] to distinguish between samples of the joint p ( ϑ , x ) with class label 1 and samples of the product of marginals p ( ϑ ) p ( x ) with class label 0, with equal label marginal probability.
For the binary cross-entropy loss, the Bayes optimal classiﬁer is
[Equation]
where σ ( · ) is the sigmoid function.
Given target parameters ϑ and an observable x supported by p ( ϑ ) and p ( x ) respectively, the learned classiﬁer ˆ d provides an approximation for the log likelihood-toevidence ratio log r ( x | ϑ ) because log r ( x | ϑ ) =
logit ( d ( ϑ , x ))
≈ logit ( ˆ d ( ϑ , x ))
=
log ˆ r ( x | ϑ ) .
The log posterior density function is approximated as log ˆ p ( ϑ | x ) = log p ( ϑ ) + log
ˆ r
( x | ϑ ) .
Balanced binary classiﬁcation for neural ratio estimation
Following Hermans et al.
[1], let us ﬁrst deﬁne the expected coverage probability of the 1 − α highest posterior density regions derived from the posterior estimator ˆ p ( ϑ | x ) as
[Equation]
where the function Θ ˆ p ( ϑ | x ) (1 − α ) yields the 1 − α highest posterior density region of ˆ p ( ϑ | x ) .
This diagnostic probes the conservativeness of the posterior estimator (or the lack thereof) and can be interpreted as the expected frequentist coverage E p ( ϑ ) E p ( x | ϑ ) (cid:2) 1 (cid:0) ϑ ∈ Θ ˆ p ( ϑ | x ) (1 − α ) (cid:1)(cid:3) .
In this work, a posterior estimator has coverage at the conﬁdence level 1 − α whenever the expected coverage probability is larger or equal to the nominal coverage probability, 1 − α .
We say that a posterior estimator is conservative when it has coverage for all conﬁdence levels.
The expected coverage probability can be plotted for various levels α , which allows to visually identify conservative posterior estimators.
The expected coverage can also be shown to be a special case of the SBC diagnostic [14] (see Appendix A), further motivating the usage of expected coverage.
Our main objective is to restrict the hypothesis space of the approximate classiﬁers ˆ d to those leading to conservative posterior estimators, hence solving the reliability concerns of NRE .
Towards this goal, we construct a hypothesis space of balanced classiﬁers and show both theoretically and empirically that they lead to posterior estimators that tend to be more conservative.
3.1 Balanced binary classiﬁcation
Deﬁnition 1.
A classiﬁer ˆ d is balanced
[Equation]
Theorem 1.
Any balanced classiﬁer ˆ d satisﬁes E p ( ϑ ,
[Equation] [Equation]
Proof.
The integral form of the balancing condition
[Equation]
implies that (cid:0) p ( x , ϑ ) + p ( ϑ ) p ( x ) (cid:1)
ˆ d ( ϑ , x ) is a valid density, both integrating to 1 and positive everywhere.
Therefore, its Kullback-Leibler (KL) divergence with p ( ϑ , x ) is positive.
Through
Jensen’s inequality, we obtain
[Equation]
Theorem 2.
Any balanced classiﬁer ˆ d satisﬁes E p ( ϑ ) p ( x )
[Equation]
Proof.
Similar to Theorem 1, see Appendix B.
Theorem 1 shows that, in expectation over the joint distribution p ( ϑ , x ) , a balanced classiﬁer ˆ d tends to make predictions whose probability values
ˆ d ( ϑ , x ) are smaller than the exact probability values
d ( ϑ , x ) .
In other words, a balanced classiﬁer ˆ d tends to be less conﬁdent than the Bayes optimal classiﬁer d .
Similarly, Theorem 2 shows that, in expectation over the product of the marginals p ( ϑ ) p ( x ) , a balanced classiﬁer tends to make predictions whose probability values 1 − ˆ d ( ϑ , x ) are smaller than the exact probability values 1 − d ( ϑ , x ) , hence showing that a balanced classiﬁer ˆ d tends to also be less conﬁdent than the Bayes optimal classiﬁer d .
We note however that these two theorems hold only in expectation, which implies that neither ˆ d ( ϑ , x ) ≤ d ( ϑ , x ) for all ϑ , x nor 1 − ˆ d ( ϑ , x ) ≤ 1 − d ( ϑ , x ) for all ϑ , x can generally be guaranteed.
Theorem 3.
The Bayes optimal classiﬁer d ( ϑ , x ) is balanced.
Proof.
Replacing the Bayes optimal classiﬁer
[Equation]
in the integral form of the balancing condition, we have ¨
[Equation]
Theorem 3 states that the Bayes optimal classiﬁer is balanced.
Therefore, minimizing the crossentropy loss while restricting the model hypothesis space to balanced classiﬁers results in the same Bayes optimal classiﬁer of Eqn. 1.
3.2 Balanced neural ratio estimation
We now extend the NRE algorithm to enforce the balancing condition.
The previous results show that enforcing the condition should result in more conservative classiﬁers ˆ d and therefore to dispersed posterior approximations.
Let us ﬁrst note that Theorem 1 can be expressed as
E p ( x ) [ E p ( ϑ | x ) [ d ( ϑ , x ) / ˆ d ( ϑ , x ]] ≥ 1 , which can (ideally) be achieved when the inner expectation is larger than 1 for all x .
In this case, the classiﬁer ˆ d will be such that ˆ d ( ϑ , x ) ≤ d ( ϑ , x ) in regions of high posterior density.
Then,
[Equation]
− d
[Equation]
ϑ , x ) , which is equivalent to ˆ r ( x | ϑ ) ≤ r ( x | ϑ ) ,
[Equation]
and ˆ p ( ϑ | x ) ≤ p ( ϑ | x ) since ˆ p ( ϑ | x ) = p ( ϑ )ˆ r ( x | ϑ ) .
Similarly, Theorem 2 implies 1 − d ( ϑ , x ) ≥
1 − ˆ d ( ϑ , x ) in regions of high prior density, which results in p ( ϑ | x ) ≤ ˆ p ( ϑ | x ) .
Between those two opposite effects, the constraint on ˆ p ( ϑ | x ) that will dominate depends on whether p ( ϑ | x ) > p ( ϑ ) or p ( ϑ | x ) < p ( ϑ ) .
If p ( ϑ | x ) > p ( ϑ ) , then ˆ p ( ϑ | x ) ≤ p ( ϑ | x ) , whereas if p ( ϑ | x ) < p ( ϑ ) then p ( ϑ | x ) ≤ ˆ p ( ϑ | x ) .
Overall, imposing the balancing condition will therefore result in approximate posteriors that lie between the prior and the exact posterior, without being more conﬁdent than they should.
Practically, the balancing condition can be targeted through a regularization penalty.
For the binary cross-entropy L [ ˆ d ] (cid:44) − E p ( ϑ , x ) [log ˆ d
( ϑ , x )] − E p ( ϑ ) p ( x ) [log(1 − ˆ d ( ϑ , x ))] and given that the balancing condition only depends on samples from p ( x ) p ( ϑ ) and p ( x , ϑ ) , the full loss functional including the balancing condition can be expressed as
[Equation]
2 , (7) where λ is a (scalar) hyper-parameter controlling the strength of the balancing condition’s contribution.
The training procedure is summarized in Algorithm 1.
Since a classiﬁer is balanced if the balancing condition cancels out, λ could, in principle, be set arbitrarily large.
However, as the balancing condition is estimated via Monte Carlo sampling, setting λ to a large value could impair the classiﬁer’s learning ability.
We found that λ = 100 works well across many problem domains with varying simulation budgets.
Inputs:
Implicit generative model p ( x | ϑ ) (simulator) and prior p ( ϑ )
Outputs:
Approximate classiﬁer
ˆ d ψ ( ϑ , x ) parameterized by ψ hyper-parameters:
Balancing condition strength λ (default = 100) and batch-size n repeat
Sample data from the joint { ϑ i , x i ∼ p ( ϑ , x ) , y i = 1 } n/ 2 i =
Sample data from the marginals { ϑ i , x i ∼ p ( ϑ ) p ( x ) , y i = 0 } ni = n/
2+
L [ ˆ d ψ ] = − 1 n (cid:80) ni
=
y i log ˆ d ψ ( ϑ i , x i )
+ (1 − y i )
log(1 − ˆ d ψ ( ϑ i , x i ))
B [ ˆ d ψ ] = 2 n (cid:80) n/ 2 i =
1 ˆ d ψ ( ϑ i , x i ) + 2 n (cid:80) ni
= n/ 2+1 ˆ d ψ
ϑ i , x i )
ψ
=
minimizer_step ( params= ψ, loss= L [ ˆ d ψ ] + λ ( B [ ˆ d ψ ] − 1) 2 ) until convergence return ˆ d ψ ( ϑ , x ) .
Experiments
We start by providing an extensive validation of BNRE on a broad range of benchmarks demonstrating that the proposed method alleviates the problem.
Section 4.2 follows up with an illustrative demonstration on the behaviour of BNRE and its hyper-parameters.
Code is available at https://github.com/
montefiore-ai/balanced-nre .
4.1 Extensive validation
Setup We evaluate the expected coverage of posterior estimators produced by both NRE and BNRE on various problems.
Those benchmarks cover a diverse set of problems from particle physics (Weinberg), epidemiology (Spatial SIR), queueing theory (M/G/1), population dynamics (Lotka
Volterra, and astronomy (Gravitational Waves).
They are representative of real scientiﬁc applications of simulation-based inference.
A more detailed description of the benchmarks can be found in
Appendix C. The architectures and hyper-parameters used for each problem are deﬁned in Appendix
D. Our evaluation considers simulation budgets of increasing size, ranging from 2 10 = 1024 to 2 17 = 131 , 072 samples, and credibility levels from 0 .
05 to 0 .
For every simulation budget, we train 5 posterior estimators for 500 epochs and determine the credible region by evaluating the approximated posterior density function in a discretized and empirically normalized grid of the parameter space with sufﬁcient resolution.
The subsequent credible region is the set of parameters whose estimated (and normalized) posterior density is higher or equal to an inclusion threshold ﬁtted to obtain the desired credibility level 1 − α .
Details on this procedure are described in Appendix E.
The expected coverage probability is estimated on 10000 unseen samples from the joint p ( ϑ , x ) , for each considered credibility level.
Expected coverage The expected coverage curves and their interpretation are detailed in Figure 1.
We observe that NRE often produces posterior estimators that are overconﬁdent, especially for small simulation budgets.
However, NRE ’s reliability increases with the availability of training data.
By contrast, BNRE produces posterior estimators that are conservative on all benchmarks for all simulation budgets .
Figure 2 explores the same phenomena through a quantity which we call the coverage AUC, highlighting the effect of the simulation budget.
Coverage AUC corresponds to the integrated signed area between the expected coverage curve and the diagonal of a particular simulation.
From this quantity it is evident there is a clear distinction between NRE and BNRE with respect to the available simulation budget.
Both methods have the tendency to converge towards 0, indicating both methods are moving closer to the Bayes optimal classiﬁer.
However, the difference between these methods lies with how this solution is approached.
While NRE can approach this limit from both sides, BNRE consistently produces coverage AUC’s above 0, corresponding to conservative posterior approximations, and therefore exhibits the desired behaviour (in expectation).
[Table]
[Figure]
[Table]
[Figure]
Statistical performance In addition to the reliability of the posteriors, we evaluate and compare the statistical performance of the posterior approximations produced by NRE and BNRE .
We estimate the expected approximate log posterior density E p ( ϑ , x ) (cid:2) log ˆ p ( ϑ | x ) (cid:3) over a large number of pairs ϑ , x .
It captures how well the posterior surrogates ˆ p ( ϑ | x ) approximate the true posteriors p ( ϑ | x ) since
[Equation]
Figure 3 shows our results.
We observe that enforcing the balancing condition for λ = 100 is associated with a loss in statistical performance.
However, the loss in statistical performance is eventually recovered by increasing the simulation budget.
In fact, practitioners might be inclined to favor reliability over statistical performance [1], although it is always a trade-off that depends on the use case.
Nevertheless, it is possible to improve the statistical performance by tuning the surrogate, or by increasing the available simulation budget as we have demonstrated.
[Figure] [Figure]
E p ( ϑ , x ) [l og ˆ p ( ϑ | x ) ]
[Figure]
( ϑ | x ) (cid:3) of the approximate log posterior density of the nominal parameters with respect to the simulation budget.
We observe that BNRE produces log posterior densities lower than NRE .
This shows that enforcing the balancing condition to have more reliable posterior approximates comes at the price of a small loss in information gain.
However, BNRE improves over the prior and eventually converges towards NRE as the simulation budget increases.
mean standard
4.2 In-depth analysis
In this section, we consider the Weinberg benchmark as described in Appendix C.
The quality of the posterior approximations produced by BNRE is initially discussed with respect to the simulation budget.
Afterwards, the effects of the hyper-parameter λ are studied.
Quality assessment Because the expected coverage does not capture the quality of an approximation in terms of information gain, we complement our assessment with a bias and variance analysis of the posterior approximations.
Let us consider the expected squared error over the approximate posterior E ˆ p ( ϑ | x ) (cid:104) ( ϑ − ϑ ∗ ) 2 (cid:105) , where ϑ ∗ is the ground truth parameter value.
With ¯ ϑ ( x ) =
E ˆ p ( ϑ | x ) [ ϑ ] , we decompose E ˆ p ( ϑ | x ) (cid:104) ( ϑ − ϑ ∗ ) 2 (cid:105) as
[Equation]
+ (cid:0) ¯ ϑ ( x ) − ϑ ∗ (cid:1) 2 .
[Figure]
observations in Figure
The expectation over the joint distribution p ( ϑ ∗ , x ) of the expected squared error can hence be decomposed in a bias term deﬁned as
[Equation]
which can be interpreted as the expected discrepancy between the nominal value ϑ ∗ and the expected posterior value ¯ ϑ .
The variance term is
[Equation]
and measures the dispersion of the posterior approximations.
Note that these terms differ from the typical statistical bias and variance of point estimators since we are considering full posterior estimators.
In particular, the bias of the Bayes optimal model does not necessarily reduce to 0 .
Figure 4 shows the evolution of expected coverage, bias and variance with respect to the available simulation budget.
By taking all plots into consideration with respect to the simulation budget, we can validate that – as suggested by theorems 1 and 2 – the increase in expected coverage is tied to an increase in variance.
However, this increase comes at the price of a slight increase in bias.
Consistent with our previous observations in Figure 3, we observe that the gap in both bias and variance reduces as the simulation budget increases.
The bias gets close to 0 for high simulation budgets, showing that the bias induced by BNRE vanishes as the simulation-budget increases.
A bias and variance analysis for all remaining benchmarks is discussed in Appendix G.
Effects of λ Finally, Figure 5 shows the effect the hyper-parameter λ on the posterior approximations, their expected coverage and the balancing condition.
BNRE is run 5 times for λ ranging fromto 2 15 and for a ﬁxed simulation budget of 1024 .
Initially, the effect on the posterior approximations is limited for small values of λ .
However, once λ increases, the balancing condition forces the posterior approximations to become increasingly dispersed and conservative.
Eventually, at least for this speciﬁc simulation budget, the posterior approximation reduces to the prior as the balancing condition becomes dominant over the cross-entropy term.
Although the global optimum remains unchanged as stated by Theorem 3, large λ values are likely to impair the training procedure.
In particular, a large λ can inﬂate the statistical noise of the Monte Carlo estimation of the balancing condition and make the classiﬁer ˆ d degenerate to a classiﬁer that is trivially balanced such as the random classiﬁer ˆ d ( ϑ , x ) = 0 .
5 for all ϑ , x .
In this case, ˆ r ( x | ϑ ) = 1 for all ϑ , x and the approximate posterior degenerates to the prior.
This effect is directly evident from Figure 5, starting from λ (cid:39) 1000 .
In practice, λ should be sufﬁciently large such that the approximate classiﬁer is balanced,
[Figure]
[Figure] [Figure]
[Figure]
E s t i m a t e d p o s t e r
i o r d e n s i t y
Eﬀect on posteriors
[Figure]
e d c
[Figure]
v e
[Figure]
ag e
[Figure] [Figure]
[Figure]
various posterior approximations in the leftmost subplot, is the fact that BNRE produces overconﬁdent and biased approximations in the presence of a small simulation budget and a small λ , indicated by their dark blue color.
However, the balancing condition can be applied to the underlying estimator to improve its reliability by increasing λ .
Ideally, λ should be as small as possible to maximize predictive performance, while at the same time remain sufﬁciently large to guarantee coverage.
From the 3 th subplot from the left, in this particular problem setting, that happens at the point where the coverage AUC transitions from being negative to positive ( λ ≈ 25 . 0 ).
while maximizing the statistical performance of the posterior estimator.
Therefore, we recommend to start with a small value for λ and to gradually increase λ until the posterior estimator becomes conservative.
We empirically found λ = 100 to be a reasonably good default value leading to good performance across all considered benchmarks with various model architectures.
Related work
In the Bayesian setting, BNRE improves the reliability of NRE by constraining the classiﬁer hypothesis space to balanced classiﬁers, which results in more conservative posteriors.
Towards the same objective of conservative and reliable approximate posteriors, Hermans et al.
[1] have shown empirically that ensembling posterior estimators increases their expected coverage.
Since the two solutions are complementary, we suggest that ensembling BNRE is a safe practice to follow.
To the best of our knowledge, no other related work exists to make Bayesian simulation-based inference algorithms more conservative and reliable.
In the frequentist setting, Cranmer et al.
[6] make use of neural ratio estimation to learn likelihood ratio test statistics.
They show that the classiﬁer ˆ d does not need to be exact for the statistic to remain the most powerful, provided that the approximate likelihood ratio is monotonic with exact likelihood ratio.
When this is not the case, robust inference remains possible by calibrating the classiﬁer, at the price of a loss in statistical power.
Similarly, for frequentist likelihood-free inference, Dalmasso et al.
[22] use classiﬁers to estimate likelihood ratio statistics and propose a procedure for guaranteeing valid hypothesis tests and conﬁdence sets.
Finally, Dalmasso et al.
[23] propose a practical procedure for the Neyman construction of conﬁdence sets with ﬁnite-sample guarantees of nominal coverage as well as diagnostics that estimate conditional coverage over the entire parameter space.
In this work, we make the assumption that the simulator is well-speciﬁed, in the sense that it accurately models the real data generation process.
However, this assumption is often violated.
To overcome this issue, Generalized Bayesian inference (GBI) extends Bayesian inference by replacing the likelihood term by with arbitrary loss function [24].
Those loss functions can be designed to mitigate speciﬁc types of misspeciﬁcations and enable robust inference, even with intractable likelihoods [25–27].
Power likelihood losses have also been shown to increase robustness to model misspeciﬁcation [28].
It consists in raising the likelihood to a power to control the impact it has over the prior.
The lower the power of likelihood, the lower the importance given to the data and the higher the uncertainty of the posterior.
It can either be set based on practitioner knowledge or derived from observed data [29].
Following the same objective, Miller and Dunson [30] introduce coarsened posteriors that condition on a neighborhood of the empirical data distribution rather than on the data itself.
This neighborhood is derived from a distance function that, when set to the relative entropy, allows the approximation of coarsened posteriors by a power posterior.
Recently, Dellaporta et al.
[31] applied
Bayesian non-parametric learning to SBI, making inference with misspeciﬁed simulator models both robust and computationally efﬁcient.
Conclusions and future work
In this work, we introduced Balanced Neural Ratio Estimation ( BNRE ), a variation of neural ratio estimation designed to produce more conservative posterior estimators, even when the likelihood-toevidence ratio estimator is not computationally faithful.
We provide theoretical arguments suggesting that enforcing the balancing condition should lead to more conservative posteriors without sacriﬁcing exactness in the large simulation budget regime.
Our theoretical results are experimentally validated on benchmarks of varying complexity.
Nevertheless, our inference algorithm comes with limitations that practitioners should keep in mind.
First, we emphasize that theorems 1 and 2 hold only in expectation, which means that we cannot provide any guarantee at the level of single inferences.
Second, the balancing condition is enforced through a regularization penalty that is not estimated exactly.
This implies that the classiﬁer ˆ d is rarely strictly balanced, although close to be, in which case theorems 1 and 2 do not hold.
Third, the beneﬁts of BNRE remain to be assessed in high-dimensional parameter spaces.
In particular, the posterior density must be evaluated on a discretized grid over the parameter space to compute credibility regions, which currently prohibits the accurate computation of expected coverage in the high-dimensional setting.
In conclusion, BNRE should not be viewed as a way to obtain conservative posterior estimators with 100% reliability, but rather as a way to increase the reliability of the posterior estimators with minimal effort and no computational overhead .
Looking forward, the balancing condition could potentially be applied to other simulation-based inference algorithms.
Future works could include a generalization to neural posterior estimation (NPE).
In fact, the likelihood-to-evidence ratio can be extracted from an approximate posterior by removing its dependence on the prior, log ˆ r ( x | ϑ ) = log ˆ p ( ϑ | x ) − log p ( ϑ ) , which in turn can be expressed as a classiﬁer ˆ d ( ϑ , x ) =
σ (log ˆ r ( x | ϑ )) on which the balancing condition can be evaluated and enforced.
Although our work focuses on amortized approximate inference, the balancing condition could also be applied to sequential inference algorithms to increase their reliability.
Finally, although our initial motivation is framed within the ﬁeld of simulation-based inference, our theoretical results are directly applicable to any binary classiﬁcation task by replacing the joint and marginal distributions in the balancing condition with the distributions of the two considered classes.
Therefore, it provides an easy-to-implement modiﬁcation for high-risk classiﬁcation problems.
Acknowledgments and Disclosure of Funding
Arnaud Delaunoy, Joeri Hermans and Antoine Wehenkel would like to thank the National Fund for
Scientiﬁc Research (F.R.S.-FNRS) for their scholarships.
Gilles Louppe is recipient of the ULiègeNRB Chair on Big Data and is thankful for the support of NRB.
Computational resources have been provided by the Consortium des Équipements de Calcul Intensif (CÉCI), funded by the National
Fund for Scientiﬁc Research (F.R.S.-FNRS) under Grant No. 2.
A Expected coverage as a special case of simulation-based calibration Simulation-
based calibration (SBC) [14] provides a way to diagnose the faithfulness of an approximate posterior distribution ˆ p ( θ | x ) .
Given an observation x ∗ ∼ p ( x ) , Talts et al.
[14] prove that, for any one-dimensional statistic f : Θ (cid:55)→ R , the rank statistic
[Equation]
) of posterior samples
ϑ ∗ ∼ p ( ϑ | x ∗ ) is uniformly distributed over the interval [0 , 1] .
Consequently, any deviation from the uniform distribution for the approximate rank statistic
[Equation]
11) indicates some error in the approximate posterior ˆ p
( ϑ | x ∗ ) .
As this holds for any statistic f , it also holds for f ( ϑ ) =
ˆ p ( ϑ | x ∗ ) .
In this special case, if ˆ r ( ϑ ∗ ) =
α , a proportion 1 − α of samples ϑ ∼ ˆ p ( ϑ | x ∗ ) have an approximate posterior density larger than ϑ ∗ .
In other words, it means that ϑ ∗ resides within the 1 − α highest posterior density region Θ ˆ p ( ϑ | x ∗ ) (1 − α ) of ˆ p ( ϑ | x ∗ ) .
Therefore, we have
[Equation]
12) and since ˆ r ( ϑ ∗ ) should be uniformly distributed, P (ˆ r ( ϑ ∗ ) ≥ α ) should be equal to 1 − α .
In practice, this test cannot be performed locally for a given x ∗ as we cannot sample from the unknown posterior distribution p ( ϑ | x ∗ ) .
Instead, SBC checks globally that ˆ r ( ϑ ∗ ) is uniformly distributed over pairs ( ϑ ∗ , x ∗ ) ∼ p ( ϑ , x ) sampled from the joint distribution, which, in the special case f ( ϑ )
=
ˆ p ( ϑ | x ∗ ) , comes down to check that
[Equation] [Equation]
is satisﬁed for all α ∈ [0 , 1] .
We recognize here the expected coverage diagnostic used in Hermans et al.
[1] and this work.
B Proof of Theorem
Theorem 2.
Any balanced classiﬁer ˆ d satisﬁes E p ( ϑ ) p ( x )
[Equation]
Proof.
From the integral form of the balancing condition, we have
[Equation]
which implies that (cid:0) p ( x , ϑ )+ p ( ϑ ) p ( x ) (cid:1)(cid:0)
1 − ˆ d ( ϑ , x ) (cid:1) is a valid density, integrating to 1 and positive everywhere.
Therefore, its Kullback-Leibler divergence with p ( ϑ ) p ( x ) is positive and, using Jensen’s inequality, we have
[Table]
C Benchmarks
The SLCP simulator models a ﬁctive problem with 5 parameters.
The observable x is composed of 8 scalars which represent the 2D-coordinates of 4 points.
The coordinate of each point is sampled from the same multivariate Gaussian whose mean and covariance matrix are parametrized by ϑ .
We consider an alternative version of the original task [4] by inferring the marginal posterior density ofof those parameters.
In contrast to its original formulation, the likelihood is not tractable due to the marginalization.
The Weinberg problem [32] concerns a simulation of high energy particle collisions e + e − → µ + µ − .
The angular distributions of the particles can be used to measure the Weinberg angle x in the standard model of particle physics.
From the scattering angle, we are interested in inferring Fermi’s constant ϑ .
The Spatial SIR model [1] involves a grid-world of susceptible, infected, and recovered individuals.
Based on initial conditions and the infection and recovery rate ϑ , the model describes the spatial evolution of an infection.
The observable x is a snapshot of the grid-world after some ﬁxed amount of time.
M/G/1 [33] models a processing and arrival queue.
The problem is described by 3 parameters ϑ that inﬂuence the time it takes to serve a customer, and the time between their arrivals.
The observable x is composed of 5 equally spaced quantiles of inter-departure times.
The Lotka-Volterra population model [34, 35] describes a process of interactions between a predator and a prey species.
The model is conditioned on 4 parameters ϑ which inﬂuence the reproduction and mortality rate of the predator and prey species.
We infer the marginal posterior of the predator parameters from time series representing the evolution of both populations over time.
The speciﬁc implementation is based on a Markov Jump Process as in Papamakarios et al.
Gravitational Waves (GW) are ripples in space-time emitted during events such as the collision of two black-holes.
They can be detected through interferometry measurements x and convey information about celestial bodies, unlocking new ways to study the universe.
We consider inferring the masses ϑ of two black-holes colliding through the observation of the gravitational wave as measured by LIGO ’s dual detectors [36, 37].
D Architectures and hyper-parameters
Table 1 summarizes the architectures and hyper-parameters used for each benchmark.
The classiﬁer architectures are separated into two parts: the embedding and the head networks.
The embedding network φ compresses the observable into a set of features.
The head network f then uses those features φ ( x ) concatenated with the parameters ϑ to predict the class, ˆ d ( ϑ , x ) =
f ( ϑ , φ ( x )) .
The learning rate is scheduled during training.
Table 1 provides the initial learning rates.
Those are then divided by 10 each time no improvement was observed on the validation loss for 10 epochs.
Further details can be found in the code repository attached to this manuscript.
[Table] [Table]
[Table] [Table]
E Estimation of the expected coverage probability
We describe in this section the methodology used to estimate the expected coverage probability
[Equation]
We consider n test simulations ( ϑ ∗ i , x i ) ∼ p ( ϑ ) p ( x | ϑ ) and compute their associated approximate posteriors ˆ p ( ϑ | x i ) in a discretized and empirically normalized grid of the parameter space.
The associated credible region is the highest density credible region, i
[Figure]
e. a credible region of the form
[Equation]
The threshold γ is computed using a dichotomic search to produce a credible region of level 1 − α .
We then estimate the empirical expected coverage probability by the proportion of nominal parameters ϑ ∗ i that falls in their associated credible region Θ ˆ p ( ϑ | x i ) (
[Equation]
− α ) ,
[Equation]
F Standard deviations of Coverage AUCs
Figure 6 shows the coverage AUC for various simulation budgets.
The mean and standard deviation over 5 runs are reported.
[Figure]
diagonal line and has a coverage AUC of 0 , as shown on the left subplot.
A conservative estimator on the other hand has a coverage AUC larger thanand an overconﬁdent estimator smaller than 0 .
We observe that while NRE can produce coverage
AUC both below or above 0 , BNRE always produces a coverage AUC larger than 0 , implying that its posterior approximations are conservative.
Solid lines represent the mean over 5 runs and shaded areas represent the standard deviation.
G Complete bias and variance analysis
Figure 7 shows the evolution of the bias and variance w.r.t.
the simulation budget on a wide variety of benchmarks.
We observe that observations made on Weinberg in Section 4 generalize to all benchmarks.
The variance obtained with BNRE is always higher or equal than the one obtained with
NRE as suggested by Theorems 1 and 2.
In addition, as suggested by Theorem 3, the bias and variance obtained with BNRE converges, as NRE , to the Bayes optimal solution.
[Figure]