Original: Contrastive Neural Ratio Estimation
Revised: Contrastive Neural Ratio Estimation

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Likelihood-to-evidence ratio estimation is usually cast as either a binary ( NRE - A )or a multiclass ( NRE - B ) classification task.
Revised: Likelihood-to-evidence ratio estimation is usually cast as either a binary ( NRE - A )or a multiclass ( NRE - B ) classification task.

Original: In contrast to the binary classificationframework, the current formulation of the multiclass version has an intrinsic andunknown bias term, making otherwise informative diagnostics unreliable.
Revised: In contrast to the binary classificationframework, the current formulation of the multiclass version has an intrinsic andunknown bias term, making otherwise informative diagnostics unreliable.

Original: Wepropose a multiclass framework free from the bias inherent to NRE - B at optimum,leaving us in the position to run diagnostics that practitioners depend on.
Revised: Wepropose a multiclass framework free from the bias inherent to NRE - B at optimum,leaving us in the position to run diagnostics that practitioners depend on.

Original: It alsorecovers NRE - A in one corner case and NRE - B in the limiting case.
Revised: It alsorecovers NRE - A in one corner case and NRE - B in the limiting case.

Original: For faircomparison, we benchmark the behavior of all algorithms in both familiar andnovel training regimes: when jointly drawn data is unlimited, when data is fixedbut prior draws are unlimited, and in the commonplace fixed data and parameterssetting.
Revised: For faircomparison, we benchmark the behavior of all algorithms in both familiar andnovel training regimes: when jointly drawn data is unlimited, when data is fixedbut prior draws are unlimited, and in the commonplace fixed data and parameterssetting.

Original: Our investigations reveal that the highest performing models are distantfrom the competitors ( NRE - A , NRE - B ) in hyperparameter space.
Revised: Our investigations reveal that the highest performing models are distantfrom the competitors ( NRE - A , NRE - B ) in hyperparameter space.

Original: We make arecommendation for hyperparameters distinct from the previous models
Revised: We make arecommendation for hyperparameters distinct from the previous models.

Original:  Introduction
Revised:  Introduction

Original: existing We begin with a motivating example: Considerthe task of inferring the mass of an exoplanet θ ofrom the light curve observations x o of a distantstar.
Revised: We begin with a motivating example: Considerthe task of inferring the mass of an exoplanet θ ofrom the light curve observations x o of a distantstar.

Original: We design a computer program that mapshypothetical mass θ to a simulated light curvex using relevant physical theory.
Revised: We design a computer program that mapshypothetical mass θ to a simulated light curvex using relevant physical theory.

Original: Our simulatorcomputes x from θ , but the inverse mapping isunspecified and likely intractable.
Revised: Our simulatorcomputes x from θ , but the inverse mapping isunspecified and likely intractable.

Original: Simulationbased inference ( SBI ) puts this problem in aprobabilistic context [13, 61].
Revised: Simulationbased inference ( SBI ) puts this problem in aprobabilistic context [12, 51].

Original: Although we cannot analytically evaluate it, we assume that thesimulator is sampling from the conditional probability distribution p ( x | θ ) .
Revised: Although we cannot analytically evaluate it, we assume that thesimulator is sampling from the conditional probability distribution p ( x | θ ) .

Original: After specifying aprior p ( θ ) , the inverse amounts to estimating theposterior p ( θ | x o ) .
Revised: After specifying aprior p ( θ ) , the inverse amounts to estimating theposterior p ( θ | x o ) .

Original: This problem setting occursacross scientific domains [1, 7, 10, 11, 29] whereθ generally represents input parameters of thesimulator and x the simulated output observation.
Revised: This problem setting occursacross scientific domains [1, 6, 9, 10, 25].

Original: Our design goal is to produce a surrogatemodel ˆ p ( θ | x ) approximating the posterior forany data x while limiting excessive simulation.

Original: Density estimation [5, 51, 52] can fit the likelihood [2, 15, 43, 53] or  posterior [6, 21, 42,50] directly; however, an appealing alternative for practitioners is estimating a ratio between distributions [12, 16, 30, 34, 65]. Specifically, the likelihood-to-evidence ratio p ( θ | x ) p ( θ ) = p ( x | θ ) p ( x )= p ( θ , x ) p ( θ ) p ( x ) .
Revised: Density estimation [42, 43] can fit the likelihood[2, 14, 36, 44] or the posterior [5, 20, 35, 41]directly; however, an appealing alternative forpractitioners is estimating a ratio between distributions [11, 29], in particular p ( θ | x ) p ( θ ) , which equals the likelihood-to-evidence ratio p ( x | θ ) p ( x )

Original: Unlike the other methods,ratio estimation enables easy aggregation of independent and identically drawn data x , an importance sampling-based diagnostic, and estimating the mutual information, see Appendix D
Revised: Unlike the other methods, ratio estimation enables easy aggregation of independentand identically drawn data x and an importance sampling-based diagnostic [26]. It also enables anestimate of the mutual information.

Original: Estimating p ( x | θ ) p ( x ) can be formulated as a binary classification task [30], where the classifier σ ◦ f w ( θ , x ) distinguishes between pairs( θ , x ) sampled either from the joint distributionp ( θ , x ) or the product of its marginals p ( θ ) p ( x ) .
Revised: Estimating p ( x | θ ) p ( x ) can be formulated as a binary classification task [26], where the classifier σ ◦ f w ( θ , x ) distinguishes between pairs( θ , x ) sampled either from the joint distributionp ( θ , x ) or the product of its marginals p ( θ ) p ( x ) .

Original: We call it NRE - A .
Revised: We call it NRE - A .

Original: The optimal classifier has
Revised: The optimal classifier has

Original: [Equation]
Revised: [Equation]

Original: Here, σ represents the sigmoid function, ◦ implies function composition, and f w is a neuralnetwork with weights w .

Original: As a part of an effortto unify different SBI methods and to improvesimulation-efficiency, Durkan et al.
Revised: In an effort to improve simulation-efficiency, Durkan et al.

Original: [16] reformulated the classification task to identify whichof K possible θ k was responsible for simulatingx .
Revised: [15] reformulated the classificationtask to identify which of K possible θ k wasresponsible for simulating x .

Original: We refer to it as NRE - B .
Revised: We refer to it as NRE - B .

Original: At optimum
Revised: At convergence

Original: [Equation]
Revised: [Equation]

Original: where an additional bias, c w ( x ) , appears.
Revised: where an additional bias, c w ( x ) , appears.

Original: g w represents another neural network, see Section 2.1.
Revised: Details are shown in Section 2.1 and Appendix A.

Original: The c w ( x ) term nullifies many of the advantages ratio estimation offers.
Revised: The c w ( x ) term nullifies many of the advantages ratio estimation offers.

Original: Practitioners wishing toperform necessary verification of their SBI results can no longer perform an importance samplingbased diagnostic.
Revised: Practitioners wishing to perform necessary verification of their SBI results can no longer perform an importancesampling-based diagnostic.

Original: c w ( x ) can be arbitrarily pathological in x , meaning that  the normalizing constantcan take on extreme values.
Revised: c w ( x ) can be arbitrarily pathological in x , meaning that computing the normalizing constant can be difficult,aggregated posteriors are unnormalized, and mutual information approximations are biased.

Original: Details about SBI are in Appendix A. There is a distinction in deep learning-based SBI betweenamortized and sequential algorithms which produce surrogate models that estimate any posteriorp ( θ | x ) or a specific posterior p ( θ | x o ) respectively.
Revised: Details about SBI are in Appendix A. There is a distinction in deep learning-based SBI betweenamortized and sequential algorithms which produce surrogate models that estimate any posteriorp ( θ | x ) or a specific posterior p ( θ | x o ) respectively.

Original: Amortized algorithms sample parameters fromthe prior, while sequential algorithms use an alternative proposal distribution–increasing efficiency atthe expense of flexibility.
Revised: Amortized algorithms sample parameters fromthe prior, while sequential algorithms use an alternative proposal distribution–increasing efficiency atthe expense of flexibility.

Original: Tejero-Cantero et al.
Revised: Tejero-Cantero et al.

Original: [64] implement both kinds of SBI models.
Revised: [54] implement both kinds of SBI models.

Original: Contribution We design a more general formulation of likelihood-to-evidence ratio estimationas a multiclass problem in which the bias inherent to NRE - B is discouraged by the loss functionand it does not appear at optimum.
Revised: Contribution We design a more general formulation of likelihood-to-evidence ratio estimationas a multiclass problem in which the bias inherent to NRE - B is discouraged by the loss functionand it does not appear at optimum.

Original: Figure 1 diagrams the interpolated performance as a function ofhyperparameters.
Revised: Figure 1 diagrams the interpolated performance as a function ofhyperparameters.

Original: It shows which settings recover NRE - A and NRE - B , also indicating that highestperformance occurs with settings distant from these.
Revised: It shows which settings recover NRE - A and NRE - B , also indicating that highestperformance occurs with settings distant from these.

Original: Figure 2 shows the relationship of the lossfunctions.
Revised: Figure 2 shows the relationship of the lossfunctions.

Original: We call our framework NRE - C 1 and expound the details in Section 2.
Revised: We call our framework NRE - C 1 and expound the details in Section 2.

Original: An existing importance sampling diagnostic [30] tests whether a classifier can distinguish p ( x | θ )samples from from samples from p ( x ) weighted by the estimated ratio.

Original: We demonstrate that ourproposed NRE - C passes this diagnostic while NRE - B does not.

Original: We evaluate NRE - B and NRE - C in a fair comparison in several training regimes in Section 3.
Revised: We evaluate NRE - B and NRE - C in a fair comparison in several training regimes in Section 3.

Original: Weperform a hyperparameter search on three simulators with tractable likelihood  by benchmarking thebehavior when (a) jointly drawn pairs ( θ , x ) are unlimited or when jointly drawn pairs ( θ , x ) arefixed
Revised: Weperform a hyperparameter search on three simulators with tractable likelihood [37] by benchmarkingthe behavior when (a) jointly drawn pairs ( θ , x ) are unlimited or when jointly drawn pairs ( θ , x ) arefixed

Original: but we (b) can draw from the prior p ( θ ) without limit or (c) are restricted to the initial pairs.
Revised: but we (b) can draw from the prior p ( θ ) without limit or (c) are restricted to the initial pairs.

Original: Wealso perform the SBI benchmark of Lueckmann et al.
Revised: Wealso perform the SBI benchmark of Lueckmann et al.

Original: [44] with our recommended hyperparameters.
Revised: [37] with our recommended hyperparameters.

Original:  Methods
Revised:  Methods

Original: The ratio between probability distributions can be estimated using the “likelihood ratio trick” bytraining a classifier to distinguish samples [12, 19, 27, 30, 47, 62, 65].
Revised: The ratio between probability distributions can be estimated using the “likelihood ratio trick” bytraining a classifier to distinguish samples [11, 18, 23, 26, 40, 52, 55].

Original: We first summarize theloss functions of NRE - A and NRE - B which approximate the intractable likelihood-to-evidence ratior ( x | θ ) :
Revised: We first summarize theloss functions of NRE - A and NRE - B which approximate the intractable likelihood-to-evidence ratior ( x | θ ) :

Original: = p ( x | θ ) p ( x ) .
Revised: = p ( x | θ ) p ( x ) .

Original: We then elaborate on our proposed generalization, NRE - C .
Revised: We then elaborate on our proposed generalization, NRE - C .

Original: Finally, we explainhow to recover NRE - A and NRE - B within our framework and comment on the normalization properties.
Revised: Finally, we explainhow to recover NRE - A and NRE - B within our framework and comment on the normalization properties.

Original: NRE - A
Revised: NRE - A

Original: Hermans et al.
Revised: Hermans et al.

Original: [30] train a binary classifier to distinguish ( θ , x ) pairs drawn dependentlyp ( θ , x ) from those drawn independently p ( θ ) p ( x ) .
Revised: [26] train a binary classifier to distinguish ( θ , x ) pairs drawn dependentlyp ( θ , x ) from those drawn independently p ( θ ) p ( x ) .

Original: This classifier is parameterized by a neuralnetwork f w which approximates log r ( x | θ ) .
Revised: This classifier is parameterized by a neuralnetwork f w which approximates log r ( x | θ ) .

Original: We seek optimal network weights
Revised: We seek optimal network weights

Original: [Equation]
Revised: [Equation]

Original: θ ( b ) , x ( b ) ∼ p ( θ ) p ( x ) and θ ( b ′ ) , x ( b ′ ) ∼ p
Revised: θ ( b ) , x ( b ) ∼ p ( θ ) p ( x ) and θ ( b ′ ) , x ( b ′ ) ∼ p

Original: θ , x ) over B samples.
Revised: θ , x ) over B samples.

Original: NRE - A ’s ratio estimate convergesto f w ∗ = log p ( x | θ ) p ( x ) given unlimited model flexibility and data.
Revised: NRE - A ’s ratio estimate convergesto f w ∗ = log p ( x | θ ) p ( x ) given unlimited model flexibility and data.

Original: Details can be found in Appendix A.
Revised: Details can be found in Appendix A.

Original: NRE - B
Revised: NRE - B

Original: Durkan et al.
Revised: Durkan et al.

Original: [16] train a classifier that selects from among K parameters ( θ 1 , . .
Revised: [15] train a classifier that selects from among K parameters ( θ 1 , . .

Original: . , θ K )which could have generated x , in contrast with NRE - A ’s binary possibilities. One of these parametersθ k is always drawn jointly with x .
Revised: . , θ K )which could have generated x . In contrast with NRE - A , one of these parameters θ k is alwaysdrawn jointly with x .

Original: The classifier is parameterized by a neural network g w whichapproximates log r ( x | θ ) .
Revised: The classifier is parameterized by a neural network g w which approximateslog r ( x | θ ) .

Original: Training is done over B samples by finding
Revised: Training is done over B samples by finding

Original: [Equation]
Revised: [Equation]

Revised: arg

Original:  where θ ( b ′ ) 1 , . .
Revised: [Equation] w [Equation] where θ ( b ′ ) 1 , . .

Original: , θ ( b ′ )
Revised: , θ ( b ′ )

Original: K ∼ p
Revised: K ∼ p

Original: ( θ ) and x ( b ′ )
Revised: ( θ ) and x ( b ′ )

Original: ∼ p ( x | θ ( b ′ ) k ) .
Revised: ∼ p ( x | θ ( b ′ ) k ) .

Original: Given unlimited model flexibility and data
Revised: Given unlimited model flexibility and data

Original: NRE - B ’s ratio estimate converges to g w ∗ ( θ , x )
Revised: NRE - B ’s ratio estimate converges to g w ∗ ( θ , x )

Original: = log p ( θ | x ) p ( θ ) + c w ∗ ( x ) .
Revised: = log p ( θ | x ) p ( θ ) + c w ∗ ( x ) .

Original: Details are in Appendix A.
Revised: Details are in Appendix A.

Original: 2.1 Contrastive Neural Ratio Estimation
Revised: 2.1 Contrastive Neural Ratio Estimation

Original: Our proposed algorithm NRE - C trains a classifier to identify which θ among K candidates isresponsible for generating a given x , inspired by NRE - B .
Revised: Our proposed algorithm NRE - C trains a classifier to identify which θ among K candidates isresponsible for generating a given x , inspired by NRE - B .

Original: We added another option that indicates xwas drawn independently, inspired by NRE - A .
Revised: We added another option that indicates xwas drawn independently, inspired by NRE - A .

Original: The introduction of the additional class yields a ratiowithout the specific c w ( x ) bias at convergence.
Revised: The introduction of the additional class yields a ratiowithout the specific c w ( x ) bias at convergence.

Original: Define Θ :=
Revised: Define Θ :=

Original: ( θ 1 , ..., θ K ) and conditional probaiblityp NRE - C
Revised: ( θ 1 , ..., θ K ) and conditional probaiblity

Original: ( Θ , x | y

Original: = k ) :

Original: = (cid:26) p ( θ 1 ) · · · p ( θ K ) p ( x ) k

Original: =p ( θ 1 ) · · · p ( θ K ) p ( x |

Original: [Equation]
Revised: [Equation]

Original: We set marginal probabilities p ( y = k ) :
Revised: We set marginal probabilities p ( y = k ) :

Original: = p K for all k ≥ 1 and p ( y = 0) :
Revised: = p K for all k ≥ 1 and p ( y = 0) :

Original: = p 0 .
Revised: = p 0 .

Original: Yielding therelationship p 

Revised: Yielding therelationship p

Original: = 1 − Kp K .
Revised: = 1 − Kp K .

Original: Let the odds of any pair being drawn dependently to completelyindependently be γ :=
Revised: Let the odds of any pair being drawn dependently to completelyindependently be γ :=

Original: Kp K p 0 .
Revised: Kp K p 0 .

Original: We now use Bayes’ formula to compute the conditional probability
Revised: We now use Bayes’ formula to compute the conditional probabilityp ( y = k | Θ , x [Equation] =

Original: [Equation]
Revised: [Equation]

Original:  We dropped the NRE - C subscript and substituted in γ to replace the p ( y ) class probabilities.
Revised: | [Equation] [Equation] We dropped the NRE - C subscript and substituted in γ to replace the p ( y ) class probabilities.

Original: We traina classifier, parameterized by neural network h w ( θ , x ) with weights w , to approximate (6) by
Revised: We traina classifier, parameterized by neural network h w ( θ , x ) with weights w , to approximate (6) by

Original: [Equation]
Revised: [Equation]

Original: We note that (7) still satisfies (cid:80) Kk =

Revised: We note that (7) still satisfies (cid:80) Kk =

Original: q w ( y = k
Revised: q w ( y = k

Original: | Θ , x ) = 1 , no matter the parameterization.
Revised: | Θ , x ) = 1 , no matter the parameterization.

Original: Optimization We design a loss function that encourages h w ( θ , x )
Revised: Optimization We design a loss function that encourages h w ( θ , x )

Original: = log p ( x | θ ) p ( x ) at convergence,and holds at optimum with unlimited flexibility and data.
Revised: = log p ( x | θ ) p ( x ) at convergence,and holds at optimum with unlimited flexibility and data.

Original:  We introduce the cross entropy lossℓ ( w ) := [Equation] p ( y
Revised: Proof is in Appendix B. We introduce thecross entropy loss

Original: [Equation]

Original: x )

Original: [Equation]
Revised: [Equation]

Original: y | Θ , x

Original: [Equation]

Original: [Equation]

Original: and minimize it towards w ∗ ∈ arg min w ℓ ( w ) .
Revised: and minimize it towards w ∗ ∈ arg min w ℓ ( w ) .

Original: We point out that the final term is symmetric up topermutation of Θ , enabling the replacement of the sum by multiplication with K .
Revised: On the second line, we revealed that the final termis symmetric up to permutation of Θ , enabling the replacement of the sum by multiplication with K .

Original: When γ and Kare known, p 0{+
+} = 11+ γ and p K = 1 K γ 1+ γ under our constraints.
Revised: When γ and K are known, p {+
+} = 11+ γ and p K = 1 K γ 1+ γ under our constraints.

Original: Without loss of generality, we letθ 1 , . . 
Revised: Without loss ofgenerality, we let θ 1 , . . .

Original: , θ K ∼ p ( θ ) and x ∼ p ( x | θ K ) .
Revised: , θ K ∼ p ( θ ) and x ∼ p ( x | θ K ) .

Original: An empirical estimate of the loss on B samples is therefore
Revised: An empirical estimate of the loss on Bsamples is therefore

Original: [Equation]
Revised: [Equation]

Original: In the first term, the classifier sees a completely independently drawn sample of x and Θ while θ K isdrawn jointly with x in the second term.
Revised: In the first term, the classifier sees a completely independently drawn sample of x and Θ while θ K isdrawn jointly with x in the second term.

Original: In both terms, the classifier considers K choices.
Revised: In both terms, the classifier considers K choices.

Original: In practice,we bootstrap both θ ( b ) 1 , . . .
Revised: In practice,we bootstrap both θ ( b ) 1 , . . .

Original: , θ ( b ) K and θ ( b ′ ) 1 , . . .
Revised: , θ ( b ) K and θ ( b ′ ) 1 , . . .

Original: , θ ( b ′ )
Revised: , θ ( b ′ )

Original: K − 1 from the same mini-batch and compare themto the same x , similarly to NRE - A and NRE - B .
Revised: K − 1 from the same mini-batch and compare themto the same x , similarly to NRE - A and NRE - B .

Original: All proof is in Appendix B.

Original: Recovering NRE - A and NRE - B
Revised: Recovering NRE - A and NRE - B

Original: NRE - C is general because specific hyperparameter settings recover
Revised: NRE - C is general because specific hyperparameter settings recover

Original: NRE - A and NRE - B .
Revised: NRE - A and NRE - B .

Original: To recover NRE - A one should set γ = 1 and K = 1 in (9) yieldingˆ [Equation]
Revised: To recover NRE - A one should set γ = 1 and K = 1 in (9) yielding

Original: [Equation]
Revised: [Equation]

Original: where we dropped the lower index.
Revised: where we dropped the lower index.

Original: Recovering NRE - B requires taking the limit γ → ∞ in the lossfunction.
Revised: Recovering NRE - B requires taking the limit γ → ∞ in the lossfunction.

Original: In that case, the first term goes to zero, and second term converges to the softmax function.
Revised: In that case, the first term goes to zero, and second term converges to the softmax function.

Original: [Equation]
Revised: [Equation]

Original: is determined by substitution into (9).
Revised: comes from substitution into (9).

Original: Both equations are obviously the same as their counterparts.
Revised: Both equations are obviously the same as their counterparts.

Original: Estimating a normalized posterior In the limit of infinite data and infinite neural network capacity(width, depth) the optimal classifier trained using NRE - C (with γ ∈ R + ) satisfies the equality:
Revised: Estimating a normalized posterior In the limit of infinite data and infinite neural network capacity(width, depth) the optimal classifier trained using NRE - C (with γ ∈ R + ) satisfies the equality:

Original: [Equation]
Revised: [Equation]

Original: In particular, we have that the following normalizing constant is trivial:
Revised: In particular, we have that the following normalizing constant is trivial:

Original: [Equation]
Revised: [Equation]

Original: This is a result of Lemma 1 in Appendix B.
Revised: This is a result of Lemma 1 in Appendix B.

Original: However, practitioners never operate in this setting, ratherthey use finite sample sizes and neural networks with limited capacity that are optimized locally.
Revised: However, practitioners never operate in this setting, ratherthey use finite sample sizes and neural networks with limited capacity that are optimized locally.

Original: Thenon-optimal function exp( h w ( θ , x )) does not have a direct interpretation as a ratio of probabilitydistributions, rather as the function to weigh the prior p ( θ ) to approximate the unnormalized posterior.
Revised: Thenon-optimal function exp( h w ( θ , x )) does not have a direct interpretation as a ratio of probabilitydistributions, rather as the function to weigh the prior p ( θ ) to approximate the unnormalized posterior.

Original: In other words, we find the following approximation for the posterior p ( θ | x ) :
Revised: In other words, we find the following approximation for the posterior p ( θ | x ) :

Original: [Equation]
Revised: [Equation]

Original: where in general the normalizing constant is not trivial, i.e. Z w ( x ) ̸ = 1 .
Revised: where in general the normalizing constant is not trivial, i.e. Z w ( x ) ̸ = 1 .

Original: As stated above, the NRE - C(and NRE - A ) objective encourages Z w ( x ) to converge to 1 .
Revised: As stated above, the NRE - C(and NRE - A ) objective encourages Z w ( x ) to converge to 1 .

Original: This is in sharp contrast to NRE - B , whereeven at optimum with an unrestricted function class a non-trivial x -dependent bias term can appear.
Revised: This is in sharp contrast to NRE - B , whereeven at optimum with an unrestricted function class a non-trivial x -dependent bias term can appear.

Original: There is no restriction on how pathological the NRE - B bias c w ( x ) can be.
Revised: There is no restriction on how pathological the NRE - B bias c w ( x ) can be.

Original: Consider a minimizer of(4), the NRE - B loss function, h w ∗ + c w ∗ ( x ) .
Revised: Consider a minimizer of(4) , the NRE - B loss function, h w ∗ + c w ∗ ( x ) .

Original: Adding any function d ( x ) cancels out in the fraction andis also a minimizer of (4).
Revised: Adding any function d ( x ) cancels out in the fraction andis also a minimizer of (4).

Original: This freedom complicates any numerical computation of the normalizingconstant and renders the importance sampling diagnostic from Section 2.2 generally inapplicable.
Revised: This freedom complicates any numerical computation of the normalizingconstant and renders the importance sampling diagnostic from Section 2.2 generally inapplicable.

Original: We report Monte Carlo estimates of Z w ( x ) on a test problem across hyperparameters in Figure 13.

Original: 2.2 Measuring performance & ratio estimator diagnostics
Revised: 2.2 Measuring performance & ratio estimator diagnostics

Original: SBI is difficult to verify because, for many use cases, the practitioner cannot compare surrogatep w ( θ | x ) to the intractable ground truth p
Revised: SBI is difficult to verify because, for many use-cases, the practitioner cannot compare results to theintractable ground truth posterior.

Original: ( θ | x ) .

Original: Incongruous with the practical use case for SBI ,much of the literature has focused on measuring the similarity between surrogate and posteriorusing two-samples tests on tractable problems.
Revised: Incongruous with practice, much of the literature has focused onmeasuring the SBI estimates by on tractable problems.

Original: For comparison with literature, we first referenceatwo-sample exactness metric which requires a tractable posterior.
Revised: We first reference a sample-based exactnessmetric.

Original: We then discuss diagnostics whichdo not require samples from p ( θ | x ) , commenting on the relevance for each NRE algorithm withempirical results.
Revised: We then discuss diagnostics which do not require a ground truth, commenting on the relevancefor each NRE algorithm with empirical results.

Original: In Appendix D, we find that a known variational bound to the mutual information istractable to estimate within SBI , and that it bounds the average Kullback-Leibler divergence betweensurrogate and posterior.

Original: We use it to evaluate various models fairly at every epoch in Figure 5.

Original: Comparing to a tractable posterior with estimates of exactness Quite a number of assessmentsof approximate posterior quality are available when samples can be drawn from both the ground truthposterior θ ∼ p ( θ |
Revised: Comparing to a tractable posterior with estimates of exactness Quite a number of assessmentsof approximate posterior quality are available when samples can be drawn from both the ground truthposterior θ ∼ p ( θ |

Original: x ) and the approximation θ ∼ q ( θ | x ) .
Revised: x ) and the approximation θ ∼ q ( θ | x ) .

Original: In the deep learning-based SBI literature,exactness is measured as a function of computational cost, usually simulator calls.
Revised: In the deep learning-based SBI literature,exactness is measured as a function of computational cost, usually simulator calls.

Original: We investigate thesimulation-efficiency of NRE - C in Section 3.3.
Revised: We investigate thesimulation-efficiency of NRE - C in Section 3.3.

Original: Based on the recommendations of Lueckmann et al.
Revised: Based on the recommendations of Lueckmann et al.

Original: [44] our experimental results are measured usingthe Classifier Two-Sample Test (C2ST) [17, 40, 41].
Revised: [37] our experimental results are measured usingthe Classifier Two-Sample Test (C2ST) [16, 33, 34].

Original: A classifier is trained to distinguish samplesfrom either the surrogate or the ground truth posterior.
Revised: A classifier is trained to distinguish samplesfrom either the surrogate or the ground truth posterior.

Original: An average classification probability onholdout data of 1.0 implies that samples from each distribution are easily identified; 0.5 implies eitherthe distributions are the same or the classifier does not have the capacity to distinguish them.
Revised: An average classification probability onholdout data of 1.0 implies that samples from each distribution are easily identified; 0.5 implies eitherthe distributions are the same or the classifier does not have the capacity to distinguish them.

Original: Importance sampling diagnostic
Revised: Importance sampling diagnostic

Original: An accurate likelihood-to-evidence weight transforms the datadistribution into the likelihood by p ( x | θ )
Revised: An accurate likelihood-to-evidence weight transforms the datadistribution into the likelihood by p ( x | θ )

Original: =
Revised: =

Original: p ( x ) r ( x | θ ) .
Revised: p ( x ) r ( x | θ ) .

Original: Since NRE necessitates simulator access,we can test the ratio estimator by training a classifier to distinguish unweighted p ( x | θ ) samples fromweighted p ( x )ˆ r ( x | θ ) samples, where ˆ r implies an estimate.
Revised: Since NRE necessitates simulator access,we can test the ratio estimator by training a classifier to distinguish unweighted p ( x | θ ) samples fromweighted p ( x )ˆ r ( x | θ ) samples, where ˆ r implies an estimate.

Original: Indistinguishability between samplesimplies either that the approximate ratio is accurate for parameter θ or that the classifier does nothave sufficient power to find predictive features.
Revised: Indistinguishability between samplesimplies either that the approximate ratio is accurate for parameter θ or that the classifier does not havesufficient power to find predictive features.

Original: Issues with classification power can be detected byassessing the classifier’s ability to distinguish p ( x ) from p ( x | θ ) .
Revised: Issues with classification power can be detected by assessing the classifier’s ability to distinguish p ( x ) from p ( x | θ ) .

Original: The performance can be visualizedin a receiver operating curve ( ROC ) or measured by the area under the curve ( ROC AUC ).
Revised: The performance can be visualized in areceiver operating curve ( ROC ) or measured by the area under the curve ( ROC AUC ).

Original: Thisdiagnostic has been used for ratio estimators before [12, 30] but it comes from training models undercovariate shift [59].
Revised: This diagnostichas been used for ratio estimators before [11, 26] but it comes from training models under covariateshift [49].

Original: It is particularly appealing because it does not require samples from p ( θ | x ) .
Revised: It is particularly appealing because it does not require ground truth posterior samples.

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: P o s t
Revised: P o s t

Original: e r
Revised: e r

Original: i o r d e n s i t y
Revised: i o r d e n s i t y

Original: [Figure]
Revised: [Figure]

Original: NRE
Revised: NRE

Original: [Figure]
Revised: [Figure]

Original:  NRE
Revised: C NRE

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original:  pp r o x i m [Figure] t e li k e li
Revised: A pp r o x i m a t e li k e li

Original: [Figure]
Revised: [Figure]

Original: oo d d e n s i t y , [Figure] a r i o u s
Revised: oo d d e n s i t y , v a r i o u s

Original: [Figure]
Revised: [Figure]

Original: failedto distinguish producesand passes the diagnostic.
Revised: failed produces passes diagnostic

Original: Durkan et al.
Revised: Durkan et al.

Original: [16] do not mention this diagnostic in their paper, but due to its intrinsic bias NRE - Bdoes not fulfill the identity necessary for this diagnostic to hold at convergence.
Revised: [15] do not mention this diagnostic in their paper, but due to its intrinsic bias NRE - Bdoes not fulfill the identity necessary for this diagnostic to hold at convergence.

Original: The unknownfactor that depends on x implies p ( x | θ ) ̸ = p ( x ) exp ◦ g w ( x | θ ) .
Revised: The unknownfactor that depends on x implies p ( x | θ ) ̸ = p ( x ) exp ◦ g w ( x | θ ) .

Original: We provide empirical evidence ofthis issue in Figure 3.
Revised: We provide empirical evidence ofthis issue in Figure 3.

Original: Although NRE - B accurately approximates the true posterior, it demonstrablyfails the diagnostic.
Revised: Although NRE - B accurately approximates the true posterior, it demonstrablyfails the diagnostic.

Original: Given the limited options for verification of SBI results, this presents a majorproblem by significantly limiting the trustworthiness of NRE - B on any problem with an intractableposterior.
Revised: Given the limited options for verification of SBI results, this presents a majorproblem by significantly limiting the trustworthiness of NRE - B on any problem with an intractableposterior.

Original: In Appendix B, we show that the unrestricted nature of the NRE - B -specific c w ( x ) biasmeans approximating p ( x | θ ) with normalized importance weights is ill-posed, i.e., the problem isnot solved by estimating the partition function.
Revised: In Appendix B, we show that the unrestricted nature of the NRE - B -specific c w ( x ) biasmeans approximating p ( x | θ ) with normalized importance weights is ill-posed.

Original: Empirical expected coverage probability For a candidate distribution to qualify as the posterior,integrating over data must return the prior.
Revised: Empirical expected coverage probability For a candidate distribution to qualify as the posterior,integrating over data must return the prior.

Original: A measurement that follows from calibration to the prioris called expected coverage probability.
Revised: A measurement that follows from calibration to the prioris called expected coverage probability.

Original: Expected coverage probability can be estimated with samplesfrom p ( θ , x ) and any amortized SBI method.
Revised: Expected coverage probability can be estimated with samplesfrom p ( θ , x ) and any amortized SBI method.

Original: Although important, ability to compute this metric doesnot distinguish NRE - C .
Revised: Although important, ability to compute this metric doesnot distinguish NRE - C .

Original: We refer the interested reader to Hermans et al.
Revised: We refer the interested reader to Hermans et al.

Original: We note that popularsequential techniques generally render this diagnostic inapplicable, with exceptions [10, 45, 46].
Revised: We note that popularsequential techniques generally render this diagnostic inapplicable, with exceptions [9, 38, 39].

Original:  Experiments
Revised:  Experiments

Original: We perform experiments in three settings to measure the exactness of surrogate models under varioushyperparameters and training regimes.
Revised: We perform experiments in three settings to measure the exactness of surrogate models under varioushyperparameters and training regimes.

Original: Section 3.1 aims to identify whether data or architecture is thebottleneck in accuracy by drawing from the joint with every mini-batch.
Revised: Section 3.1 aims to identify whether data or architecture is thebottleneck in accuracy by drawing from the joint with every mini-batch.

Original: The next experiments aim todetermine how to optimally extract inference information given a limited amount of simulation data.
Revised: The next experiments aim todetermine how to optimally extract inference information given a limited amount of simulation data.

Original: Section 3.2 leverages a cheap prior by drawing new contrastive parameters with every mini-batch.
Revised: Section 3.2 leverages a cheap prior by drawing new contrastive parameters with every mini-batch.

Original: Finally, Section 3.3 applies the commonplace training regime for deep learning-based SBI literatureof fixed data and bootstrapped contrastive parameters.
Revised: Finally, Section 3.3 applies the commonplace training regime for deep learning-based SBI literatureof fixed data and bootstrapped contrastive parameters.

Original: In this setting, we also benchmark NRE - C onten inference tasks from Lueckmann et al.
Revised: In this setting, we also benchmark NRE - C onten inference tasks from Lueckmann et al.

Original: [44] using our recommended hyperparameters.
Revised: [37] using our recommended hyperparameters.

Original: On all hyperparameter searches we consider three simulators from the simulation-based inferencebenchmark, namely SLCP, Two Moons, and Gaussian Mixture [44].
Revised: On all hyperparameter searches we consider three simulators from the simulation-based inferencebenchmark, namely SLCP, Two Moons, and Gaussian Mixture [37].

Original: SLCP is a difficult inference taskwhich has a simple likelihood and complex posterior [21, 53].
Revised: SLCP is a difficult inference taskwhich has a simple likelihood and complex posterior [20, 44].

Original: Parameters are five dimensional andthe data are four samples from a two dimensional random variable.
Revised: Parameters are five dimensional andthe data are four samples from a two dimensional random variable.

Original: Two Moons is two dimensional with a crescent-shaped bimodal posterior.
Revised: Two Moons is two dimensional with a crescent-shaped bimodal posterior.

Original: Finally, the Gaussian Mixture data draws from a mixtureof two, two-dimensional normal distributions with extremely different covariances [3, 60, 61, 66].
Revised: Finally, the Gaussian Mixture data draws from a mixtureof two, two-dimensional normal distributions with extremely different covariances [3, 50, 51, 56].

Original: [Figure]
Revised: [Figure]

Original: Our surrogate models are parameterized by oneof two neural network architectures.
Revised: Our surrogate models are parameterized by oneof two neural network architectures.

Original: Small NNconforms to the benchmark, up to the use ofbatch normalization, and has 50 hidden units andtwo residual blocks, Large NN has 128 hiddenunits and three residual blocks.
Revised: Small NNconforms to the benchmark, up to the use ofbatch normalization, and has 50 hidden units andtwo residual blocks, Large NN has 128 hiddenunits and three residual blocks.

Original: We comparetheir performance on a grid of γ and K values.
Revised: We comparetheir performance on a grid of γ and K values.

Original: We report post-training results using the C2ST,and mid-training validation loss for insight intoconvergence rate.
Revised: We report post-training results using the C2ST,and mid-training validation loss for insight intoconvergence rate.

Original: We generally use residualnetworks [28] with batch normalization [33] andtrain them using adam [37].
Revised: We generally use residualnetworks [24] with batch normalization [28] andtrain them using adam [32].

Original: We also run NRE - Bwith the same architecture for comparison.
Revised: We also run NRE - Bwith the same architecture for comparison.

Original: Tocompare with NRE - B we set the number of totalcontrastive parameters equal.
Revised: Tocompare with NRE - B we set the number of totalcontrastive parameters equal.

Original: What does fair comparison mean in our experimentation?
Revised: What does fair comparison mean in our experimentation?

Original: We compare models across fixednumber of gradient steps.
Revised: We compare models across fixednumber of gradient steps.

Original: This implies that models with more classes, i.e., greater K , evaluatethe classifier-in-training on more pairs at a giventraining step than a model with fewer classes.
Revised: This implies that models with more classes, i.e., greater K , evaluatethe classifier-in-training on more pairs at a giventraining step than a model with fewer classes.

Original: An alternative which we do not apply in thispaper: Vary the number of gradient steps, holding the number of pair evaluations fixed, i.e., amodel with higher K sees the same number ofpairs as a model with lower K but the modelwith lower K has been updated more times.
Revised: An alternative which we do not apply in thispaper: Vary the number of gradient steps, holding the number of pair evaluations fixed, i.e., amodel with higher K sees the same number ofpairs as a model with lower K but the modelwith lower K has been updated more times.

Original: Weleave this analysis for future work.
Revised: Weleave this analysis for future work.

Original: 3.1 Behavior with unlimited data
Revised: 3.1 Behavior with unlimited data

Original: What is responsible for inaccuracies of the surrogate model when training has saturated?
Revised: What is responsible for inaccuracies of the surrogate model when training has saturated?

Original: (a)the amount of training data (b) the flexibility ofthe model?
Revised: (a)the amount of training data (b) the flexibility ofthe model?

Original: In this section we provide new simulation and parameter data with every mini-batchand train until saturation, thereby eliminating the possibility of (a).
Revised: In this section we provide new simulation and parameter data with every mini-batchand train until saturation, thereby eliminatingthe possibility of (a).

Original: To produce the contrastiveparameters,we bootstrap the newly drawn  Θ , this is explained in Appendix A.
Revised: To produce the contrastiveparameters, the newly drawn mini-batch parameters Θ are bootstrapped.

Original: The setting is similar to REJ - ABC where simulations are drawn until the posterior has converged.
Revised: The setting is similar to REJ - ABC where simulations are drawn until the posterior has converged.

Original: Theresults of this study will provide a baseline to compare with limited-data results and help us understandhow the deep learning architecture’s limitations are affected by our introduced hyperparameters.
Revised: Theresults of this study will provide a baseline to compare with limited-data results and help us understandhow the deep learning architecture’s limitations are affected by our introduced hyperparameters.

Original: Theresults are reported in the top row of Figure 4.
Revised: Theresults are reported in the top row of Figure 4.

Original: The trend is that increasing the number of contrastive examples helps NRE - B and NRE - C .
Revised: The trend is that increasing the number of contrastive examples helps NRE - B and NRE - C .

Original: There isnot a clear trend with respect to γ in the C2ST.
Revised: There isnot a clear trend with respect to γ in the C2ST.

Original: Study of the detailed validation losses in Appendix Creveals that high γ is associated with higher variance validation loss during training, but here it doesnot seem to strongly affect convergence rate.
Revised: Study of the detailed validation losses in Appendix Creveals that high γ is associated with higher variance validation loss during training, but here it doesnot seem to strongly affect convergence rate.

Original: Saturation is reached before the maximum number ofepochs have elapsed and Large NN converges to a better result.
Revised: Saturation is reached before the maximum number ofepochs have elapsed and Large NN converges to a better result.

Original: Although obfuscated in the averagedresults in Figure 4, the result is obvious on SLCP, the most difficult inference task that provides anopportunity for a more flexible architecture to improve.
Revised: Although obfuscated in the averagedresults in Figure 4, the result is obvious on SLCP, the most difficult inference task which provides anopportunity for a more flexible architecture to improve.

Original: We argue that generally the performance bottleneck has to do with the number of contrastiveparameters, network flexibility, or training details rather than the amount of data.
Revised: We argue that generally the performance bottleneck has to do with the number of contrastiveparameters, network flexibility, or training details rather than the amount of data.

Original: Despite saturatingvalidation losses, the C2ST improves based on these factors.
Revised: Despite saturatingvalidation losses, the C2ST improves based on these factors.

Original: Intuitively a  flexible model can continueto extract information from this unlimited set of jointly drawn ( θ , x ) .
Revised: Intuitively a more flexible model couldcontinue to extract information from this unlimited set of jointly drawn ( θ , x ) pairs.

Original: This has consequences becausenetwork flexibility may limit performance in the benchmark case.
Revised: This has someconsequence because network flexibility may limit performance in the benchmark case.

Original: See Appendix C.
Revised: Appendix Ccontains more detailed results.

Original: 3.2 Leveraging fast priors (drawing theta)
Revised: 3.2 Leveraging fast priors (drawing theta)

Original: In practice, the simulator is often slow but one can often draw from the prior at the same pace astraining a neural network for one mini-batch.
Revised: In practice, the simulator is often slow but one can often draw from the prior at the same pace astraining a neural network for one mini-batch.

Original: Our goal is to understand how our hyperparameter areaffected by this setting and whether it is valuable to use this technique in practice.
Revised: Our goal is to understand how our hyperparameter areaffected by this setting and whether it is valuable to use this technique in practice.

Original: To our knowledge,this training regime has not been explored in the deep learning-based SBI literature.
Revised: To our knowledge,this training regime has not been explored in the deep learning-based SBI literature.

Original: Initially we draw a fixed set of around 20,000 samples ( θ , x ) ∼ p
Revised: Initially we draw a fixed set of around 20,000 samples ( θ , x ) ∼ p

Original: θ , x ) .
Revised: θ , x ) .

Original: For every mini-batch duringtraining, we sample all necessary contrastive parameters from the prior.
Revised: For every mini-batch duringtraining, we sample all necessary contrastive parameters from the prior.

Original: For each term in ˆ ℓ γ,K wetake the same batch of contrastive parameters and reshuffle them, thereby equalizing the numberof samples seen by NRE - B and NRE - C , up to bootstrap.
Revised: For each term in ˆ ℓ γ,K wetake the same batch of contrastive parameters and reshuffle them, thereby equalizing the numberof samples seen by NRE - B and NRE - C , up to bootstrap.

Original: The averaged C2ST results from the threeinference tasks are reported in the middle row of Figure 4.
Revised: The averaged C2ST results from the threeinference tasks are reported in the middle row of Figure 4.

Original: The resulting estimators are markedly less sensitive to the number of contrastive parameters than in
Revised: The resulting estimators are markedly less sensitive to the number of contrastive parameters than in

Original: Section 3.1.
Revised: Section 3.1.

Original: The number of network parameters has a positive effect, although again this is best seenon the hardest task, SLCP, in the Appendix C. Empirically, lower values of γ improve performancewith Large NN; however, there is still high variance and this result may be from noise.
Revised: The number of network parameters has a positive effect, although again this is best seenon the hardest task, SLCP, in the Appendix C. Empirically, lower values of γ improve performance;however, there is still high variance and this result may be from noise.

Original: In thissetting, γ ̸ = 1 has a very small negative effect on convergence rate compared to unity, as seen in thevalidation loss plots in Appendix C. Since training has saturated, we claim this result implies drawingcontrastive parameters from the prior helps extract the maximum amount of information from fixedsimulation data x –without being strongly affected by the number of contrastive parameters K .[Figure] I ( 0 ) w ( ; x ) NRE-B
Revised: In this setting, γ ̸ = 1 hasa very small negative effect on convergence rate compared to unity, as seen in the validation lossplots in Appendix C. Since training has saturated, we claim this result implies drawing contrastiveparameters from the prior helps extract the maximum amount of information from fixed data–withoutbeing strongly affected by the number of contrastive parameters K .

Original: [Figure]

Original: [-B 10101010 3 Epochs2 
-]
Revised: [-B 10101010 3 Epochs2 
-]

Original: I ( 0 ) w ( ; x ) (gamma) = 0.

Original: 10101010 3 Epochs (gamma) = 0.

Original: 10101010 3 Epochs (gamma) = 0.

Original: 10101010 3 Epochs (gamma) = 1. 10101010 3 Epochs (gamma) = 10. 10101010 3 Epochs (gamma) = 100. Contrastiveparameters
Revised: 3.3 Simulation-based inference benchmark

Original: [Figure]
Revised: [Figure]

Original: 3.3 Simulation-based inference benchmark

Original: In this section, we assume the traditional literature setting of limited simulation and prior budget.
Revised: In this section, we assume the traditional literature setting of limited simulation and prior budget.

Original: Once we’ve selected hyperparameters based on a grid search of a subset of the SBI benchmark, weperform the entire benchmark with those hyperparameters.
Revised: Once we’ve selected hyperparameters based on a grid search of a subset of the SBI benchmark, weperform the entire benchmark with those hyperparameters.

Original: Hyperparameter search To compare to the previous two sections, the amount of training data wasfixed such that each epoch was comparable to Section 3.1, this amounts to about 20,000 samples.
Revised: Hyperparameter search To compare to the previous two sections, the amount of training data wasfixed such that each epoch was comparable to Section 3.1, this amounts to about 20,000 samples.

Original: Wefirst inspect the C2ST results in the last row of Figure 4.
Revised: Wefirst inspect the C2ST results in the last row of Figure 4.

Original: The sensitivity to K appears less than in
Revised: The sensitivity to K appears less than in

Original: Section 3.1 but more than in Section 3.2.
Revised: Section 3.1 but more than in Section 3.2.

Original: Smaller γ slightly improves Large NN’s performance butthe noise makes this result uncertain.
Revised: Similarly to the other sections, the effects of γ on the C2STare not very predictable.

Original: The larger network performs better–best seen in Appendix C.
Revised: The larger network performs better, but this is best seen in Appendix C.

Original: We do a Monte Carlo estimate a mutual information bound, denoted − ˆ I (0) w ( θ ; x ) , to show performance on the SLCP task as a function of epochs in Figure 5. For both NRE - B and NRE - C , increasing
Revised: A visualization of the effects of K and γ on convergence rate is available in Figure 5, but note thatvalidation loss is not comparable across models so the plots do not indicate relative posterior accuracy.

Original:  K tends to positively affect the convergence rate.
Revised: Increasing K has a positive effect on the convergence rate.

Original: In NRE - C , γ < 1  slows convergence and slightlyincreases performance compared to unity; while γ > 1 slows convergence and reduces performance.
Revised: γ < 1 definitely slows down convergencecompared to unity; the effects of γ > 1 are unclear from this data.

Original: Our take-away recommendation is that γ ∈ [0 . 1 , 10] has a small effect, but generally γ = 1 has thefastest convergence rate with smaller γ performing slightly better with flexible networks.
Revised: Our take-away recommendation is that γ ∈ [0 . 1 , 10] has a small effect, but generally γ = 1 hasthe fastest convergence rate and the effect on the accuracy of the estimated posterior is not sopredictable.

Original: Generally,we saw improved performanceon the C2ST by increasing contrastive parameters K .
Revised: Generally, we saw improved performance by increasing contrastive parameters K .

Original: However,larger K also increases the magnitude of the normalizing constant, see Appendix D. Since webootstrap parameters, we note the natural upper limit given a mini-batch of B parameters, namely
Revised: Sincewe bootstrap parameters, we note the natural upper limit given a mini-batch of B parameters, namely

Original: B/ 2 compares the maximum contrastive parameters to each x without reusing any θ .
Revised: B/ 2 compares the maximum contrastive parameters to each x without reusing any θ .

Original: Benchmark We performed the benchmark tasks from Lueckmann et al.
Revised: Benchmark We performed the benchmark tasks from Lueckmann et al.

Original: [44] with NRE - C .
Revised: [37] with NRE - C .

Original: Ourarchitecture and hyperparameters deviate based on the results of the last paragraph.
Revised: Ourarchitecture and hyperparameters deviate based on the results of the last paragraph.

Original: We applied thelargest number of computationally practical contrastive parameters, namely K = 99 , and set γ = 1 .
Revised: We applied thelargest number of computationally practical contrastive parameters, namely K = 99 , and set γ = 1 .

Original: Since we identified potential architecture bottlenecks in Sections 3.1 and 3.3, we used Large NN forour results.
Revised: Since we identified potential architecture bottlenecks in Sections 3.1 and 3.3, we used Large NN forour results.

Original: The averaged results are visible in Table 1.
Revised: The averaged results are visible in Table 1.

Original: Note that we trained a collection of amortizedestimators then predicted the posterior for each of the ten observations x 1 , . . .
Revised: Note that we trained a collection of amortizedestimators then predicted the posterior for each of the ten observations x 1 , . . .

Original: , x 10 per inferenceproblem, whereas Lueckmann et al.
Revised: , x 10 per inferenceproblem, whereas Lueckmann et al.

Original: [44] trained an estimator for every x .
Revised: [37] trained an estimator for every x .

Original: NRE - C performed better than NRE - B at all budgets and generally well among the amortized algorithms.(Those are ones without an S prefix.)
Revised: NRE - C performed better than NRE - B at all budgets and generally well among the amortized algorithms.(Those are ones without an S prefix.)

Original: The high performance at 10 5 simulation budget implies that ourdesign has high capacity and may scale well.
Revised: The high performance at 10 5 simulation budget implies that ourdesign has high capacity and may scale well.

Original: Sequential methods naturally lead to high exactnesssince they are tailored to the specific observation x o , but they come with drawbacks since they cannotpractically perform empirical expected coverage tests [31, 46] nor bound the mutual information, aswe do in Figure 5.
Revised: Sequential methods naturally lead to high exactnesssince they are tailored to the specific observation x o , but they come with drawbacks since they cannotpractically perform empirical expected coverage tests [27, 39], as mentioned in Section 2.2.

Original: Detailed results per task can be found in Appendix C.
Revised: Detailedresults per task can be found in Appendix C.

Original:  Conclusion
Revised:  Conclusion

Original: [Table]
Revised: [Table]

Original: We introduced a generalization of NRE - A and
Revised: We introduced a generalization of NRE - A and

Original: NRE - B which discourages the NRE - B specificc w ( x ) bias term and produces an unbiased estimate of the likelihood-to-evidence ratio at optimum.
Revised: NRE - B which discourages the NRE - B specificc w ( x ) bias term and produces an unbiased estimate of the likelihood-to-evidence ratio at optimum.

Original: This property has implications for theimportance sampling diagnostic, which is critical to practitioners.
Revised: This property has implications for theimportance sampling diagnostic, which is critical to practitioners.

Original: We used a variational boundon the mutual information as a model-selectionmetric that allowed us to compare models withdifferent hyperparameters. Further discussion in Appendix D. In the context of NRE - C , we found that increasing K improved the C2ST in most cases, atthe price of increasing the normalizing constant.
Revised: Furthermore, we found thatincreasing the number of contrastive parameters

Original: Setting γ ≈ 1 was generally the fastest to saturate and lower values seemed to slightly improveperformance with the Large NN and fixed simulation data.
Revised: K helped in most cases and setting γ ≈ 1 wasgenerally the fastest to saturate.

Original: These results indicate that better performance can be achieved by using NRE - C with thesehyperparameters than the other version of NRE .
Revised: These resultsindicate that better performance can be achievedby using NRE - C with these hyperparameters thanthe other version of NRE .

Original: When both training parameters θ and simulation datax are unlimited, the architecture size plays an important role in the quality of surrogate model.
Revised: We performed experiments which indicated that the architectureplays an important role in the quality of surrogate model when the data is unlimited.

Original: Wetried drawing contrastive parameters from the prior, a commonly available practical use-case, andfound that it damped the effect of K on the C2ST.
Revised: We trieddrawing contrastive parameters from the prior, acommonly available practical use-case, and found that it damped the importance of K .

Original: Generally, for highest accuracy with a fixed budgetset Klarge, up to the mini-batch size divided by 2, and γ ≈ 1 . 0 for the best convergence rate.
Revised: Generally, werecommend using large K up to the mini-batch size divided by 2 and γ ≈ 1 . 0 for the best convergencerate.

Original: Whenthe prior is sampled for every mini-batch, applying a higher K may be less valuable. 
Revised: When the prior admits sampling at the rate of training a mini-batch, applying a higher K maybe less valuable. Discussions about estimating the mutual information are found in Appendix C.1. Broader Impact The societal implications of SBI are similar to other inference methods.

Original: In situationswhere the normalizing constant should be very close to unity, i.e., for practitioners who want to rundiagnostics on their results, drawing θ from the prior with every mini-batch and using a small K islikely best as it provided good accuracy with small values for the normalizing constant.
Revised: Themethods are primarily scientific and generally lead to interesting discoveries; however, one must becareful to use an accurate generative model and to carefully test empirical results. Model mismatchand untested inference can lead to incorrect conclusions. That is why we emphasize the importanceof the diagnostics in our paper. This nuance can be missed by practitioners who are making decisionsabout matters that affect living creatures.

Original: Checklist
Revised: Checklist

Original: The checklist follows the references.
Revised: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information onhow to answer these questions.
Revised: Please read the checklist guidelines carefully for information onhow to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
Revised: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .

Original: You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
Revised: You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.

Original: For example:
Revised: For example:

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [Yes] See Section .•
Revised: [Yes] See Section .•

Original: Did you include the license to the code and datasets?
Revised: Did you include the license to the code and datasets?

Original: [No] The code and the data areproprietary.•
Revised: [No] The code and the data areproprietary.•

Original: Did you include the license to the code and datasets?
Revised: Did you include the license to the code and datasets?

Original: [N/A]
Revised: [N/A]

Original: Please do not modify the questions and only use the provided macros for your answers.
Revised: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the
Revised: Note that the

Original: Checklist section does not count towards the page limit.
Revised: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
Revised: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes] We introduce the generalization in Section 2.1.We
Revised: [Yes] We introduce the generalization in Section 2.1.

Original: show that it has the properties we claim in Appendix B.
Revised: Weshow that it has the properties we claim in Appendix B.

Original: We perform the experiments in
Revised: We perform the experiments in

Original: Section 3 and make a hyperparameter recommendation in Section
Revised: Section 3 and make a hyperparameter recommendation in Section

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] We discussed the behavior of
Revised: [Yes] We discussed the behavior of

Original: ours and other algorithms in the finite sample setting, see Section 2.1 and limitations ofexisting to test the convergence of our method, see Section 2.2.(c)
Revised: ours and other algorithms in the finite sample setting, see Section 2.1 and limitations ofexisting to test the convergence of our method, see Section 2.2.(c)

Original: Did you discuss any potential negative societal impacts of your work?
Revised: Did you discuss any potential negative societal impacts of your work?

Original: [Yes] We briefly mentioned them in Section 4.
Revised: [Yes] We brieflymentioned them in Section 4.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes] This work applies toy simulation data thus no ethics concerns were raised
Revised: [Yes] This work applies toy simulation data thus no ethics concerns were raised

Original: by the paper.
Revised: by the paper.

Original: Other matters of ethics during experimentation and writing conformed.
Revised: Other matters of ethics during experimentation and writing conformed.

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] These are laid
Revised: [Yes] These are laid

Original: out in Appendix B.
Revised: out in Appendix B.

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] We have only one real
Revised: [Yes] We have only one real

Original: theoretical result and that is proven in Lemma 1.
Revised: theoretical result and that is proven in Lemma 1.

Original: Anything else follows algebraically
Revised: Anything else follows algebraically

Original: and is shown in detail.
Revised: and is shown in detail.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [Yes] The code will
Revised: [Yes] The code will

Original: be made available via a link in the supplemental material.
Revised: be made available via a link in the supplemental material.

Original: It is a python package with
Revised: It is a python package with

Original: example calls and self-explanatory installation instructions.
Revised: example calls and self-explanatory installation instructions.

Original: The details will be shown
Revised: The details will be shown

Original: in Section C. Results will be released upon acceptance.
Revised: in Section C. Results will be released upon acceptance.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters,
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters,

Original: how they
Revised: how they

Original: were chosen)?
Revised: were chosen)?

Original: [Yes] Additionally to the hyperparameter search which is the primary
Revised: [Yes] Additionally to the hyperparameter search which is the primary

Original: experimental result, seen in Section 3, we included more details in Appendix C.
Revised: experimental result, seen in Section 3, we included more details in Appendix C.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experi-
Revised: (c) Did you report error bars (e g., with respect to the random seed after running experi-

Original: ments multiple times)?
Revised: ments multiple times)?

Original: [Yes] The hyperparameter searches indeed included error bars
Revised: [Yes] The hyperparameter searches indeed included error bars

Original: in Section 3.
Revised: in Section 3.

Original: We did not include error bars in the computation of the diagnostic since
Revised: We did not include error bars in the computation of the diagnostic since

Original: we emphasize that it is made possible by our method, not its statistical properties.
Revised: we emphasize that it is made possible by our method, not its statistical properties.

Original: The
Revised: The

Original: details of the benchmark, including some uncertainty in the last experimental section
Revised: details of the benchmark, including some uncertainty in the last experimental section

Original: are shown in Appendix C.
Revised: are shown in Appendix C.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type
Revised: (d) Did you include the total amount of compute and the type of resources used (e g., type

Original: of GPUs, internal cluster, or cloud provider)?
Revised: of GPUs, internal cluster, or cloud provider)?

Original: [Yes] This is discussed in Appendix C.
Revised: [Yes] This is discussed in Appendix C.

Original: If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] We cited the
Revised: [Yes] We cited the

Original: benchmark [44] several times in the paper.
Revised: benchmark [37] several times in the paper.

Original: Also the sbi package [64]
Revised: Also the sbi package [54]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes] The license is mentioned in Ap-
Revised: [Yes] The license is mentioned in Ap-

Original: pendix A
Revised: pendix A

Original: (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes]
Revised: [Yes]

Original: I
Revised: I

Original: will link to my code in the supplemental material as discussed above.
Revised: will link to my code in the supplemental material as discussed above.

Original: Upon acceptance
Revised: Upon acceptance

Original: results will be made available.
Revised: results will be made available.

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?
Revised: using/curating?

Original: [N/A]
Revised: [N/A]

Original: The publicly available data was used in accordance with the
Revised: The publicly available data was used in accordance with the

Original: aforementioned license in [44].
Revised: aforementioned license in [37].

Original: There was no need to get consent or mention it in the
Revised: There was no need to get consent or mention it in the

Original: paper.
Revised: paper.

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?
Revised: information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: The data is generated by toy mathematical
Revised: The data is generated by toy mathematical

Original: models which abstractly relate parameters to data.
Revised: models which abstractly relate parameters to data.

Original: It is highly unlikely any culture
Revised: It is highly unlikely any culture

Original: would find them offensive.
Revised: would find them offensive.

Original: If you used crowdsourcing or conducted research with human subjects...
Revised: If you used crowdsourcing or conducted research with human subjects...

Original: (a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review
Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable?
Revised: Board (IRB) approvals, if applicable?

Original: [N/A]
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A]
Revised: [N/A]