Towards Reliable Simulation-Based Inference with Balanced Neural Ratio Estimation
Abstract
Modern approaches for simulation-based inference rely upon deep learning surrogates to enable approximate inference with computer simulators.
In practice,the estimated posteriors’ computational faithfulness is, however, rarely guaranteed.
For example, Hermans et al.
[1] show that current simulation-based inference algorithms can produce posteriors that are overconﬁdent, hence risking false inferences.
In this work, we introduce Balanced Neural Ratio Estimation ( BNRE ), a variation ofthe NRE algorithm
[2] designed to produce posterior approximations that tend to bemore conservative, hence improving their reliability, while sharing the same Bayesoptimal solution.
We achieve this by enforcing a balancing condition that increasesthe quantiﬁed uncertainty in small simulation budget regimes while still convergingto the exact posterior as the budget increases.
We provide theoretical argumentsshowing that BNRE tends to produce posterior surrogates that are more conservativethan NRE ’s. We evaluate BNRE on a wide variety of tasks and show that it producesconservative posterior surrogates on all tested benchmarks and simulation budgets.
Finally, we emphasize that BNRE is straightforward to implement over NRE anddoes not introduce any computational overhead.
Introduction
Many areas of science and engineering use parametric computer simulations to describe complexstochastic generative processes.
In this setting, Bayesian inference provides a principled frameworkto identify parameters matching empirical observations.
Computer simulations, however, deﬁnethe necessary likelihood function only implicitly, which prevents its evaluation and the use ofclassical inference algorithms.
To overcome this obstacle, recent simulation-based inference (SBI)algorithms [3] build upon deep learning surrogates to approximate parts of the Bayes rule andenable approximate inference.
For example, [4, 5] build a surrogate of the likelihood function while[6, 7, 2, 8, 9] approximate the likelihood-to-evidence ratio.
The posterior can also be targeted directlywith variational inference, as proposed by [10, 11, 5].
These algorithms can each be made amortizedor be run sequentially to drive the training towards a target observation and improve the simulationefﬁciency of the procedure [10, 12, 2, 11, 4, 8, 5].
However, sequential methods have the drawback ofbeing computationally expensive to diagnose as the surrogates are only valid for the target observation[1].
Truncated marginal neural ratio estimation [9] alleviates this issue by introducing a sequentialalgorithm that builds a surrogate valid in a local region around the target.
Since modern simulation-based inference algorithms rely on deep learning surrogates, concernsnaturally arise regarding their computational faithfulness and whether they are sufﬁciently adequatefor the inference task of interest.
In Bayesian inference, these concerns can be at least partiallyaddressed with diagnostics designed to probe the correct behaviour of the inference method, suchas ˆ R diagnostics for MCMC [13], or to assess the quality of posterior approximations directly.
The latter include diagnostics such as simulation-based calibration (SBC) [14] or coverage-baseddiagnostics [15, 1].
As discussed by Hermans et al.
[1], posterior approximations must be conservativeto guarantee reliable and meaningful inferences, even when approximations are not faithful.
Forexample, in the physical sciences, where the goal is often to constrain parameters of interest, wronglyexcluding plausible values could drive the scientiﬁc inquiry in the wrong direction, whereas failingto exclude implausible values because of (too) conservative estimations is much less detrimental.
Unfortunately, the same authors also demonstrate that current simulation-based inference algorithmscan lead to overconﬁdent surrogates and therefore false inferences.
Scientiﬁc use cases requiringconservative inference include for example the study of dark matter models in particle physics andastrophysics [16], which could be cold, warm, or hot dark matter.
In general, thermal dark mattermodels can be described by a single parameter, the dark matter thermal relic mass, which can beintuitively thought of as the energy the dark matter particle had in the Early Universe.
Small valuescorrespond to warm or hot dark matter, while high values are descriptive of cold dark matter.
Applyingan inference algorithm without diagnosing the learned estimator could lead to constraints smallerthan they should be.
For example, whenever an overconﬁdent estimator produces posterior estimatesthat favor cold dark matter models, it could simultaneously reject alternative models, such as theextensively studied Sterile Neutrino, a potential candidate for the Warm Dark Matter particle.
In this work, we develop a novel algorithm that not only converges to exact inference as the simulationbudget increases, but which is also more likely to produce conservative surrogates in small simulationbudget regimes.
Towards this objective, we propose a variant of the NRE algorithm called Balanced Neural Ratio Estimation ( BNRE ), which enforces a balancing condition on the binary neural classiﬁerto increase the reliability of its posterior approximations.
The structure of the manuscript is outlined as follows.
Section 2 describes the formalism and thenecessary background.
Section 3 describes BNRE and provides theoretical arguments towards itsconservativeness and reliability.
Section 4 illustrates our main results and provides insights regardingthe behaviour of the method.
Finally, Section 5 discusses related work while Section 6 summarizes ourcontributions and hints at future work.
Code is available at github.com/anonymous/anonymous .
Background2.1 Statistical formalism
This work is concerned with simulation-based inference algorithms that produce posterior approximations ˆ p ( ϑ | x ) under the following semantics.
Target parameters ϑ denote the parameters ofthe model
and we make the reasonable assumption that the prior p ( ϑ ) is tractable.
The model isgenerically expressed as a computer program, a simulator, that describes the forward dynamics ofinterest based on the input parameters ϑ .
The simulator implicitly deﬁnes the likelihood functionp ( x | ϑ ) .
While we cannot directly evaluate the density p ( x | ϑ ) , we can execute the computerprogram to generate synthetic observables x ∼ p ( x | ϑ ) .
Every observable x o is tied to groundtruth parameters ϑ ∗ whose forward evaluation within the simulator produced x ∗ .
Of special importance to Bayesians is the notion of a credible region , which is a domain Θ withinthe target parameter space that satisﬁes
[Equation]
for some observable x ∗ andconﬁdence level 1 − α .
Because many such regions exist, we target the credible region with thesmallest volume, also known as the highest posterior density region [17, 18].
2.2 Neural ratio estimation
Neural Ratio Estimation ( NRE ) is an established approach in the simulation-based inference literatureboth from frequentist [6] and Bayesian [7, 2, 8, 9] perspectives.
In essence, all protocols rely onthe density-ratio trick [19, 20, 6] to construct a surrogate of a likelihood ratio.
In this work, weconsider an amortized estimator ˆ r ( x | ϑ ) of the intractable likelihood-to-evidence ratio r
( x | ϑ ) =
p ( ϑ , x ) /p
( ϑ ) p ( x ) =
p ( x | ϑ ) /p
( x ) that can be learned by training a binary classiﬁer ˆ d : X × Θ (cid:55)→[0 , 1] to distinguish between samples of the joint p ( ϑ , x ) with class label 1 and samples of the productof marginals p ( ϑ ) p ( x ) with class label 0.
For the binary cross-entropy loss, the Bayes optimalclassiﬁer is
[Equation]
where σ ( · ) is the sigmoid function.
Given target parameters ϑ and an observable x supported by p ( ϑ )and p ( x ) respectively, the learned classiﬁer ˆ d provides an approximation for the log likelihood-toevidence ratio log r ( x | ϑ ) because log r ( x | ϑ ) =
logit ( d ( ϑ , x ))
≈ logit ( ˆ d ( ϑ , x ))
=
log ˆ r ( x | ϑ ) .
The log posterior density function is approximated as log ˆ p ( ϑ | x ) = log p ( ϑ ) + log
ˆ r ( x | ϑ ) .
Balanced binary classiﬁcation for neural ratio estimation
Following Hermans et al.
[1], let us ﬁrst deﬁne the expected coverage probability of the 1 − αhighest posterior density regions derived from the posterior estimator
ˆ p ( ϑ | x ) as
[Equation]
where the function Θ ˆ p ( ϑ | x ) (1 − α ) yields the 1 − α highest posterior density region of ˆ p ( ϑ | x ) .
This diagnostic probes the conservativeness of the posterior estimator (or the lack thereof) and can beinterpreted as the expected frequentist coverage E p ( ϑ ) E p ( x | [Equation] (cid:2) 1 (cid:2)
[Equation]
In this work, a posterior estimator has coverage at the conﬁdence level 1 − α whenever the expectedcoverage probability is larger or equal to the nominal coverage probability, 1 − α .
We say that aposterior estimator is conservative when it has coverage for all conﬁdence levels.
The expectedcoverage probability can be plotted for various levels α , which allows to visually identify conservativeposterior estimators.
The expected coverage can also be shown to be a special case of the SBCdiagnostic [14] (see Appendix A), further motivating the usage of expected coverage.
Our main objective is to restrict the hypothesis space of the approximate classiﬁers ˆ d to those leadingto conservative posterior estimators, hence solving the reliability concerns of NRE .
Towards this goal,we construct a hypothesis space of balanced classiﬁers and show both theoretically and empiricallythat they lead to posterior estimators that tend to be more conservative.
3.1 Balanced binary classiﬁcation
Deﬁnition 1.
A classiﬁer ˆ d is balanced if E p
[Equation]
ϑ , x
[Equation] [Equation]
Theorem 1.
Any balanced classiﬁer ˆ d satisﬁes E p ( ϑ , x )
[Equation]
Proof.
The integral form of the balancing condition
[Equation]
[Equation]
implies that (cid:0) p ( x , ϑ ) + p ( ϑ ) p ( x ) (cid:1)
ˆ d ( ϑ , x ) is a valid density, both integrating to 1 and positiveeverywhere.
Therefore, its Kullback-Leibler (KL) divergence with p ( ϑ , x ) is positive.
Through
Jensen’s inequality, we obtain
[Equation]
Theorem 2.
Any balanced classiﬁer ˆ d satisﬁes E p ( ϑ ) p ( x )
[Equation]
Proof.
Similar to Theorem 1, see Appendix B.
Theorem 1 shows that, in expectation over the joint distribution p ( ϑ , x ) , a balanced classiﬁer ˆ dtends to make predictions whose probability values ˆ d ( ϑ , x ) are smaller than the exact probabilityvalues d
( ϑ , x ) .
In other words, a balanced classiﬁer ˆ d tends to be less conﬁdent than the Bayesoptimal classiﬁer d .
Similarly, Theorem 2 shows that, in expectation over the product of the marginalsp ( ϑ ) p ( x ) , a balanced classiﬁer tends to make predictions whose probability values 1 − ˆ d ( ϑ , x ) aresmaller than the exact probability values 1 − d ( ϑ , x ) , hence showing that a balanced classiﬁer ˆ dtends to also be less conﬁdent than the Bayes optimal classiﬁer d .
We note however that these twotheorems hold only in expectation, which implies that neither ˆ d ( ϑ , x ) ≤ d ( ϑ , x ) for all ϑ , x nor1 − ˆ d ( ϑ , x ) ≤ 1 − d ( ϑ , x ) for all ϑ , x can generally be guaranteed.
Theorem 3.
The Bayes optimal classiﬁer d ( ϑ , x ) is balanced.
Proof.
Replacing the Bayes optimal classiﬁer
[Equation]
[Equation]
in the integral form of the balancing condition, we have
[Equation]
Theorem 3 states that the Bayes optimal classiﬁer is balanced.
Therefore, minimizing the crossentropy loss while restricting the model hypothesis space to balanced classiﬁers results in thesame Bayes optimal classiﬁer of Eqn. 1.
3.2 Balanced neural ratio estimation
We now extend the NRE algorithm to enforce the balancing condition.
The previous resultsshow that enforcing the condition should result in more conservative classiﬁers ˆ d and thereforeto dispersed posterior approximations.
Let us ﬁrst note that Theorem 1 can be expressed as
E p ( x ) [ E p ( ϑ | x ) [ d ( ϑ , x ) / ˆ d ( ϑ , x ]] ≥ 1 , which can (ideally) be achieved when the inner expectation is larger than 1 for all x .
In this case, the classiﬁer ˆ d will be such that ˆ d ( ϑ , x ) ≤ d ( ϑ , x ) inregions of high posterior density.
Then,
[Equation]
, which is equivalent to ˆ
[Equation]
x
[Equation]
and ˆ p ( ϑ | x ) ≤ p ( ϑ | x ) since ˆ p ( ϑ | x ) = p ( ϑ )ˆ r ( x | ϑ ) .
Similarly, because of Theorem 2, theclassiﬁer ˆ d will be such that 1 − d ( ϑ , x ) ≥ 1 − ˆ d ( ϑ , x ) in regions of high prior density, whichresults in p ( ϑ | x ) ≤ ˆ p ( ϑ | x ) .
Between those two opposite effects, the constraint on ˆ p ( ϑ | x ) thatwill dominate depends on whether p ( ϑ | x ) > p ( ϑ ) or p ( ϑ | x ) < p ( ϑ ) .
If p ( ϑ | x ) > p ( ϑ ) , thenˆ p ( ϑ | x ) ≤ p ( ϑ | x ) , whereas if p ( ϑ | x ) < p ( ϑ ) then p ( ϑ | x ) ≤ ˆ p ( ϑ | x ) .
Overall, imposing the 142 balancing condition will therefore result in approximate posteriors that lie between the prior and theexact posterior, without being more conﬁdent than they should.
Practically, the balancing condition can be targeted through a regularization penalty.
For the binarycross-entropy L [ ˆ d ] (cid:44) − E p ( ϑ , x ) [log ˆ d
( ϑ , x )] − E p ( ϑ ) p ( x ) [log(1 − ˆ d ( ϑ , x ))] and given that the balancing condition only depends on samples from p ( x ) p ( ϑ ) and p ( x , ϑ ) , the full loss functional includingthe balancing condition can be expressed as
[Equation]
where λ is a (scalar) hyper-parameter controlling the strength of the balancing condition’s contribution.
The training procedure is summarized in Algorithm 1.
Since the balancing condition needs to be 0for a classiﬁer to be balanced, the hyper-parameter controlling the strength of the balancing conditioncould, in principle, be set arbitrarily large.
However, as the balancing condition is estimated via
Monte Carlo sampling, setting λ to a large value could impair the classiﬁer’s learning ability.
Wefound that λ
= 100 works well across many problem domains with varying simulation budgets.
Inputs:
Implicit generative model p ( x | ϑ ) (simulator) and prior p ( ϑ )
Outputs:
Approximate classiﬁer
ˆ d ψ ( ϑ , x ) parameterized by ψ hyper-parameters:
Balancing condition strength λ (default = 100) and batch-size n repeat
Sample data from the joint { ϑ i , x i ∼ p ( ϑ , x ) , y i = 1 } n/ 2 i =
Sample data from the marginals { ϑ i , x i ∼ p ( ϑ ) p ( x ) , y i = 0 } ni = n/
2+
[Equation]
ψ =[-− 1 n (cid:80) ni =
 y i log ˆ d ψ ( ϑ i , x i ) + (1 − y i )-]
minimizer_step ( params
[Equation]
, loss
[Equation]
) until convergence return ˆ d ψ ( ϑ , x ) .
Experiments
We start by providing an extensive validation of BNRE on a broad range of benchmarks demonstrating that the proposed method alleviates the problem.
Section 4.2 follows up with an illustrativedemonstration on the behaviour of BNRE and its hyper-parameters.
4.1 Extensive validation
Setup We evaluate the expected coverage of posterior estimators produced by both NRE and BNREon various problems.
Those benchmarks cover a diverse set of problems from particle physics(Weinberg), epidemiology (Spatial SIR), queueing theory (M/G/1), population dynamics (Lotka Volterra, and astronomy (Gravitational Waves).
They are representative of real scientiﬁc applicationsof simulation-based inference.
A more detailed description of the benchmarks can be found in
Appendix C. The architectures and hyper-parameters used for each problem are deﬁned in Appendix D. Our evaluation considers simulation budgets of increasing size, ranging from 2 10 = 1024 to2 17 = 131 , 072 samples, and credibility levels from 0 .
05 to 0 .
For every simulation budget,we train 5 posterior estimators for 500 epochs and determine the credible region by evaluating theapproximated posterior density function in a discretized and empirically normalized grid of theparameter space with sufﬁcient resolution.
The subsequent credible region is the set of parameterswhose estimated (and normalized) posterior density is higher or equal to an inclusion threshold ﬁttedto obtain the desired credibility level 1 − α .
Details on this procedure are described in Appendix E.
The expected coverage probability is estimated on 10000 unseen samples from the joint p ( ϑ , x ) , foreach considered credibility level.
Expected coverage The expected coverage curves and their interpretation are detailed in Figure 1.
We observe that NRE often produces posterior estimators that are overconﬁdent, especially for smallsimulation budgets.
However, NRE ’s reliability increases with the availability of training data.
Bycontrast, BNRE produces posterior estimators that are conservative on all benchmarks for allsimulation budgets .
Figure 2 explores the same phenomena, highlighting the effect of the simulationbudget, by computing the integrated signed area between the expected coverage curve and thediagonal of a particular simulation which we call Coverage AUC. From this quantity it is evident thereis a clear distinction between NRE and BNRE with respect to the available simulation budget. Bothmethods have the tendency to converge towards 0, indicating both methods are moving closer to the Bayes optimal classiﬁer.
However, the difference between these methods lies with how this solutionis approached.
While NRE can approach this limit from both sides, BNRE consistently produces
[Figure] [Figure]
[Figure]
larger thanand an overconﬁdent estimator smaller than 0 .
We observe that while NRE can produce coverage
AUC both below or above 0 , BNRE always produces a coverage AUC larger than 0 , implying that its posterior approximations are conservative on average.
The means over 5 runs are reported.
A complete overview, including standard deviations, are provided in Appendix F. coverage AUC’s above 0, corresponding to conservative posterior approximations, and thereforeexhibits the desired behaviour (in expectation).
Statistical performance In addition to the reliability of the posteriors, we evaluate and compare thestatistical performance of the posterior approximations produced by NRE and BNRE .
We estimate theexpected approximate log posterior density E p ( ϑ , x ) (cid:2) log ˆ p ( ϑ | x ) (cid:3) over a large number of pairs ϑ , x ,which captures how well the posterior surrogates ˆ p ( ϑ | x ) approximate the true posteriors p ( ϑ | x )since E p
[Equation] Figure 3 shows our results.
We observe that enforcing the balancing condition for λ = 100 isassociated with a loss in statistical performance. However, the loss in statistical performance iseventually recovered by increasing the simulation budget. In fact, practitioners might be inclined to
SLCP
Weinberg
Spatial
[Figure]
M [Figure] G [Figure]
Lotka Volterra
Gravitational
[Figure]
[Figure]
budget represented by the colormap.
The bias and variance are represented on the right side of the plot.
BNRE is run with λ = 100 .
Consistent with our previous observations in Figure 3, we observe that the gap in both bias and variance reduces as the simulation budget increases.
Futhermore, in contrast with NRE , the posterior approximations of BNRE are tending towards being increasingly calibrated while at the same time being conservative.
Solid lines represent the mean over 5 runs and shaded areas represent the standard deviation.
favor reliability over statistical performance [1], although it is always a trade-off that depends on theuse case.
Nevertheless, it is possible to improve the statistical performance by tuning the surrogate,or by increasing the available simulation budget as we have demonstrated.
4.2 In-depth analysis
In this section, we consider the Weinberg benchmark as described in Appendix C.
The quality ofthe posterior approximations produced by BNRE is initially discussed with respect to the simulationbudget.
Afterwards, the effects of the hyper-parameter λ are studied.
Quality assessment Because the expected coverage does not capture the quality of an approximation in terms of information gain, we complement our assessment with a bias and variance analysis ofthe posterior approximations.
Let us consider the expected squared error over the approximate posterior E ˆ p ( ϑ | x ) (cid:104) ( ϑ − ϑ ∗ ) 2 (cid:105) , where ϑ ∗ is the ground truth parameter value.
With ¯ ϑ ( x ) =
E ˆ p ( ϑ | x ) [ ϑ ] , we decompose E ˆ p ( ϑ | x ) (cid:104) ( ϑ − ϑ ∗ ) 2 (cid:105) as
[Equation]
The expectation over the joint distribution p ( ϑ ∗ , x ) of the expected squared error can hence bedecomposed in a bias term deﬁned asbias
[Equation]
which can be interpreted as the expected discrepancy between the nominal value ϑ ∗ and the expectedposterior value ¯ ϑ .
The variance term is
[Equation]
and measures the dispersion of the posterior approximations.
Note that these terms differ fromthe typical statistical bias and variance of point estimators since we are considering full posteriorestimators.
In particular, the bias of the Bayes optimal model does not necessarily reduce to 0 .
Figure 4 shows the evolution of expected coverage, bias and variance with respect to the availablesimulation budget.
By taking all plots into consideration with respect to the simulation budget, wecan validate that – as suggested by theorems 1 and 2 – the increase in expected coverage is tied to anincrease in variance.
However, this increase comes at the price of a slight increase in bias.
Consistentwith our previous observations in Figure 3, we observe that the gap in both bias and variance reducesas the simulation budget increases.
The bias gets close to 0 for high simulation budgets, showing thatthe bias induced by BNRE vanishes as the simulation-budget increases.
A bias and variance analysisfor all remaining benchmarks is discussed in Appendix G.
Effects of λ Finally, Figure 7 in Appendix H shows the effect the hyper-parameter λ on theposterior approximations, their expected coverage and the balancing condition.
BNRE is run 5 timesfor λ ranging from 1 to 2 15 and for a ﬁxed simulation budget of 1024 .
Initially, the effect on theposterior approximations is limited for small values of λ .
However, once λ increases, the balancingcondition forces the posterior approximations to become increasingly dispersed and conservative.
Eventually, at least for this speciﬁc simulation budget, the posterior approximation reduces to theprior as the balancing condition becomes dominant over the cross-entropy term.
Although the globaloptimum remains unchanged as stated by Theorem 3, large λ values are likely to impair the trainingprocedure.
In particular, a large λ can inﬂate the statistical noise of the Monte Carlo estimation of thebalancing condition and make the classiﬁer ˆ d degenerate to a classiﬁer that is trivially balanced suchas the random classiﬁer ˆ d ( ϑ , x ) = 0 .
5 for all ϑ , x .
In this case, ˆ r ( x | ϑ ) = 1 for all ϑ , x and theapproximate posterior degenerates to the prior.
This effect is directly evident from Figure 7, startingfrom λ (cid:39) 1000 .
In practice, λ should be sufﬁciently large such that the approximate classiﬁer isbalanced, while maximizing the statistical performance of the posterior estimator.
Therefore, werecommend to start with a small value for λ and to gradually increase λ until the posterior estimatorbecomes conservative.
We empirically found λ = 100 to be a reasonably good default value leadingto good performance across all considered benchmarks with various model architectures.
Related work
In the Bayesian setting, BNRE improves the reliability of NRE by constraining the classiﬁer hypothesisspace to balanced classiﬁers, which results in more conservative posteriors.
[-Eﬀect on coverage AUC 1010 3 Hyper-parameter λ 0 .0 .0 .0 .0 .
-]
Towards the same objective of conservative and reliable approximate posteriors, Hermans et al.
[1] have shown empiricallythat ensembling posterior estimators increases their expected coverage.
Since the two solutions arecomplementary, we suggest that ensembling BNRE is a safe practice to follow.
To the best of ourknowledge, no other related work exists to make Bayesian simulation-based inference algorithmsmore conservative and reliable.
In the frequentist setting, Cranmer et al.
[6] make use of neural ratio estimation to learn likelihoodratio test statistics.
They show that the classiﬁer ˆ d does not need to be exact for the statistic to remain the most powerful, provided that the approximate likelihood ratio is monotonic with exact likelihoodratio.
When this is not the case, robust inference remains possible by calibrating the classiﬁer, at theprice of a loss in statistical power.
Similarly, for frequentist likelihood-free inference, Dalmasso et al.[22] use classiﬁers to estimate likelihood ratio statistics and propose a procedure for guaranteeingvalid hypothesis tests and conﬁdence sets.
Finally, Dalmasso et al.
[23] propose a practical procedurefor the Neyman construction of conﬁdence sets with ﬁnite-sample guarantees of nominal coverage aswell as diagnostics that estimate conditional coverage over the entire parameter space.
In this work, we make the assumption that the simulator is well-speciﬁed, in the sense that it accuratelymodels the real data generation process.
However, this assumption is often violated.
To overcome thisissue, Generalized Bayesian inference (GBI) extends Bayesian inference by replacing the likelihoodterm by with arbitrary loss function [24].
Those loss functions can be designed to mitigate speciﬁctypes of misspeciﬁcations and enable robust inference, even with intractable likelihoods [25–27].
Power likelihood losses have also been shown to increase robustness to model misspeciﬁcation [28]. It consists in raising the likelihood to a power to control the impact it has over the prior. The lowerthe power of likelihood, the lower the importance given to the data and the higher the uncertaintyof the posterior. It can either be set based on practitioner knowledge or derived from observed data[29].
Following the same objective, Miller and Dunson [30] introduce coarsened posteriors thatcondition on a neighborhood of the empirical data distribution rather than on the data itself. Thisneighborhood is derived from a distance function that, when set to the relative entropy, allows theapproximation of coarsened posteriors by a power posterior. Recently, Dellaporta et al. [31] applied
Bayesian non-parametric learning to SBI, making inference with misspeciﬁed simulator models bothrobust and computationally efﬁcient.
Conclusions and future work
In this work, we introduced Balanced Neural Ratio Estimation ( BNRE ), a variation of neural ratioestimation designed to produce more conservative posterior estimators, even when the likelihood-toevidence ratio estimator is not faithful.
We provide theoretical arguments that suggest that enforcingthe balancing condition should lead to more conservative posteriors without sacriﬁcing exactnessin the large simulation budget regime.
Our theoretical results are experimentally validated onbenchmarks of varying complexity.
Nevertheless, our inference algorithm comes with limitations that practitioners should keep in mind.
First, we emphasize that theorems 1 and 2 hold only in expectation, which means that we cannotprovide any guarantee at the level of single inferences.
Second, the balancing condition is enforcedthrough a regularization penalty that is not estimated exactly.
This implies that the classiﬁer ˆ d is rarelystrictly balanced, although close to be, in which case theorems 1 and 2 do not hold.
Third, the beneﬁtsof BNRE remain to be assessed in high-dimensional parameter spaces.
In particular, the posteriordensity must be evaluated on a discretized grid over the parameter space to compute credibility regions,which currently prohibits the accurate computation of expected coverage in the high-dimensionalsetting.
In conclusion, BNRE should not be viewed as a way to obtain conservative posteriorestimators with 100% reliability, but rather as a way to increase the reliability of the posteriorestimators with minimal effort and no computational overhead .
Looking forward, the balancing condition could potentially be applied to other simulation-basedinference algorithms.
Future works could include a generalization to neural posterior estimation(NPE).
In fact, the likelihood-to-evidence ratio can be extracted from an approximate posterior byremoving its dependence on the prior, log ˆ r ( x | ϑ ) = log ˆ p ( ϑ | x ) − log p ( ϑ ) , which in turn can beexpressed as a classiﬁer ˆ d ( ϑ , x ) =
σ (log ˆ r ( x | ϑ )) on which the balancing condition can be evaluatedand enforced.
Although our work focuses on amortized approximate inference, the balancingcondition could also be applied to sequential inference algorithms to increase their reliability.
Finally, although our initial motivation is framed within the ﬁeld of simulation-based inference, ourtheoretical results are directly applicable to any binary classiﬁcation task by replacing the joint andmarginal distributions in the balancing condition with the distributions of the two considered classes.
Therefore, it provides an easy-to-implement modiﬁcation for high-risk classiﬁcation problems.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See attached source code orrepository.
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[No] There arenone to our knowledge.(d)
Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes](b) Did you include complete proofs of all theoretical results?
[Yes]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See thesupplementary materials in attachment.(b)
Did you specify all the training details (e.g., data splits, hyper-parameters, how theywere chosen)?
[Yes] See section 4 and Appendix D.(c)
Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[No]
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A] We use none.
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] We use none.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A] We use none.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]