Original: Contrastive Neural Ratio Estimation
Revised: Contrastive Neural Ratio Estimation

Original: Abstract
Revised: Abstract

Original: Likelihood-to-evidence ratio estimation is usually cast as either a binary ( NRE - A ) or a multiclass ( NRE - B ) classification task.
Revised: Likelihood-to-evidence ratio estimation is usually cast as either a binary ( NRE - A ) or a multiclass ( NRE - B ) classification task.

Original: In contrast to the binary classification framework, the current formulation of the multiclass version has an intrinsic and unknown bias term, making otherwise informative diagnostics unreliable.
Revised: In contrast to the binary classification framework, the current formulation of the multiclass version has an intrinsic and unknown bias term, making otherwise informative diagnostics unreliable.

Original: We propose a multiclass framework free from the bias inherent to NRE - B at optimum, leaving us in the position to run diagnostics that practitioners depend on.
Revised: We propose a multiclass framework free from the bias inherent to NRE - B at optimum, leaving us in the position to run diagnostics that practitioners depend on.

Original: It also recovers NRE - A in one corner case and NRE - B in the limiting case.
Revised: It also recovers NRE - A in one corner case and NRE - B in the limiting case.

Original: For fair comparison, we benchmark the behavior of all algorithms in both familiar and novel training regimes: when jointly drawn data is unlimited, when data is fixed but prior draws are unlimited, and in the commonplace fixed data and parameters setting.
Revised: For fair comparison, we benchmark the behavior of all algorithms in both familiar and novel training regimes: when jointly drawn data is unlimited, when data is fixed but prior draws are unlimited, and in the commonplace fixed data and parameters setting.

Original: Our investigations reveal that the highest performing models are distant from the competitors ( NRE - A , NRE - B ) in hyperparameter space.
Revised: Our investigations reveal that the highest performing models are distant from the competitors ( NRE - A , NRE - B ) in hyperparameter space.

Original: We make a recommendation for hyperparameters distinct from the previous models.
Revised: We make a recommendation for hyperparameters distinct from the previous models.

Original: We suggest a bound on the mutual information as a performance metric for simulation-based inference methods, without the need for posterior samples, and provide experimental results.
Revised: We suggest a bound on the mutual information as a performance metric for simulation-based inference methods, without the need for posterior samples, and provide experimental results.

Original:  Introduction
Revised:  Introduction

Original: We begin with a motivating example: Consider the task of inferring the mass of an exoplanet θ o from the light curve observations x
Revised: We begin with a motivating example: Consider the task of inferring the mass of an exoplanet θ o from the light curve observations x

Original: o of a distant star.
Revised: o of a distant star.

Original: We design a computer program that maps hypothetical mass θ to a simulated light curve x using relevant physical theory.
Revised: We design a computer program that maps hypothetical mass θ to a simulated light curve x using relevant physical theory.

Original: Our simulator computes x from θ , but the inverse mapping is unspecified and likely intractable.
Revised: Our simulator computes x from θ , but the inverse mapping is unspecified and likely intractable.

Original: Simulationbased inference ( SBI ) puts this problem in a probabilistic context [13, 64].
Revised: Simulationbased inference ( SBI ) puts this problem in a probabilistic context [13, 64].

Original: Although we cannot analytically evaluate it, we assume that the simulator is sampling from the conditional probability distribution p ( x | θ ) .
Revised: Although we cannot analytically evaluate it, we assume that the simulator is sampling from the conditional probability distribution p ( x | θ ) .

Original: After specifying a prior p ( θ ) , the inverse amounts to estimating the posterior p ( θ | x o ) .
Revised: After specifying a prior p ( θ ) , the inverse amounts to estimating the posterior p ( θ | x o ) .

Original: This problem setting occurs across scientific domains [1, 7, 10, 11, 29] where θ generally represents input parameters of the simulator and x the simulated output observation.
Revised: This problem setting occurs across scientific domains [1, 7, 10, 11, 29] where θ generally represents input parameters of the simulator and x the simulated output observation.

Original: Our design goal is to produce a surrogate model ˆ p ( θ | x ) approximating the posterior for any data x while limiting excessive simulation.
Revised: Our design goal is to produce a surrogate model ˆ p ( θ | x ) approximating the posterior for any data x while limiting excessive simulation.

Original: Density estimation [5, 54, 55] can fit the likelihood [2, 15, 43, 56] or posterior [6, 21, 42, 53] directly; however, an appealing alternative for practitioners is estimating a ratio between distributions [12, 16, 30, 34, 69].
Revised: Density estimation [5, 54, 55] can fit the likelihood [2, 15, 43, 56] or posterior [6, 21, 42, 53] directly; however, an appealing alternative for practitioners is estimating a ratio between distributions [12, 16, 30, 34, 69].

Original: Specifically, the likelihood-to-evidence ratio p ( θ | x
Revised: Specifically, the likelihood-to-evidence ratio p ( θ | x

Original: [Equation]
Revised: [Equation]

Original: p
Revised: p

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: p
Revised: p

Original: [Equation]
Revised: [Equation]

Original: p ( x ) .
Revised: p ( x ) .

Original: Unlike the other methods, ratio estimation enables easy aggregation of independent and identically drawn data x .
Revised: Unlike the other methods, ratio estimation enables easy aggregation of independent and identically drawn data x .

Original: Ratio and posterior estimation can compute bounds on the mutual information and an importance sampling diagnostic.
Revised: Ratio and posterior estimation can compute bounds on the mutual information and an importance sampling diagnostic.

Original: Notation details are defined in Section
Revised: Notation details are defined in Section

Original: Estimating p ( x | θ ) p ( x ) can be formulated as a binary classification task [30], where the classifier σ ◦ f w ( θ , x ) distinguishes between pairs ( θ , x ) sampled either from the joint distribution p ( θ , x ) or the product of its marginals p ( θ ) p ( x ) .
Revised: Estimating p ( x | θ ) p ( x ) can be formulated as a binary classification task [30], where the classifier σ ◦ f w ( θ , x ) distinguishes between pairs ( θ , x ) sampled either from the joint distribution p ( θ , x ) or the product of its marginals p ( θ ) p ( x ) .

Original: We call it NRE - A .
Revised: We call it NRE - A .

Original: The optimal classifier has
Revised: The optimal classifier has

Original: [Equation]
Revised: [Equation]

Original: Here, σ represents the sigmoid function, ◦ implies function composition, and f w is a neural network with weights w .
Revised: Here, σ represents the sigmoid function, ◦ implies function composition, and f w is a neural network with weights w .

Original: As a part of an effort to unify different SBI methods and to improve simulation-efficiency, Durkan et al.
Revised: As a part of an effort to unify different SBI methods and to improve simulation-efficiency, Durkan et al.

Original: [16] reformulated the classification task to identify which of K possible θ k was responsible for simulating x .
Revised: [16] reformulated the classification task to identify which of K possible θ k was responsible for simulating x .

Original: We refer to it as NRE - B .
Revised: We refer to it as NRE - B .

Original: At optimum
Revised: At optimum

Original: [Equation]
Revised: [Equation]

Original: where an additional bias, c w ( x ) , appears.
Revised: where an additional bias, c w ( x ) , appears.

Original: g w represents another neural network.
Revised: g w represents another neural network.

Original: The c w ( x ) term nullifies many of the advantages ratio estimation offers.
Revised: The c w ( x ) term nullifies many of the advantages ratio estimation offers.

Original: c w ( x ) can be arbitrarily pathological in x , meaning that the normalizing constant can take on extreme values.
Revised: c w ( x ) can be arbitrarily pathological in x , meaning that the normalizing constant can take on extreme values.

Original: This limits the applicability of verification tools like the importance sampling-based diagnostic in Section 2.2.
Revised: This limits the applicability of verification tools like the importance sampling-based diagnostic in Section 2.2.

Original: The c w ( x ) term also arises in contrastive learning [23, 71] with Ma and Collins [45] attempting to estimate it in order to reduce its impact.
Revised: The c w ( x ) term also arises in contrastive learning [23, 71] with Ma and Collins [45] attempting to estimate it in order to reduce its impact.

Original: We will propose a method that discourages this bias instead.
Revised: We will propose a method that discourages this bias instead.

Original: Further discussion in Appendix D.
Revised: Further discussion in Appendix D.

Original: There is a distinction in deep learning-based SBI between amortized and sequential algorithms which produce surrogate models that estimate any posterior p ( θ | x ) or a specific posterior p ( θ | x o ) respectively.
Revised: There is a distinction in deep learning-based SBI between amortized and sequential algorithms which produce surrogate models that estimate any posterior p ( θ | x ) or a specific posterior p ( θ | x o ) respectively.

Original: Amortized algorithms sample parameters from the prior, while sequential algorithms use an alternative proposal distribution–increasing efficiency at the expense of flexibility.
Revised: Amortized algorithms sample parameters from the prior, while sequential algorithms use an alternative proposal distribution–increasing efficiency at the expense of flexibility.

Original: Amortization is usually necessary to compute diagnostics that do not require samples from p ( θ | x o ) and amortized estimators are empirically more reliable [31].
Revised: Amortization is usually necessary to compute diagnostics that do not require samples from p ( θ | x o ) and amortized estimators are empirically more reliable [31].

Original: Our study therefore focuses on amortized algorithms.
Revised: Our study therefore focuses on amortized algorithms.

Original: Contribution We design a more general formulation of likelihood-to-evidence ratio estimation as a multiclass problem in which the bias inherent to NRE - B is discouraged by the loss function and it does not appear at optimum.
Revised: Contribution We design a more general formulation of likelihood-to-evidence ratio estimation as a multiclass problem in which the bias inherent to NRE - B is discouraged by the loss function and it does not appear at optimum.

Original: Figure 1 diagrams the interpolated performance as a function of hyperparameters.
Revised: Figure 1 diagrams the interpolated performance as a function of hyperparameters.

Original: It shows which settings recover NRE - A and NRE - B , also indicating that highest performance occurs with settings distant from these.
Revised: It shows which settings recover NRE - A and NRE - B , also indicating that highest performance occurs with settings distant from these.

Original: Figure 2 shows the relationship of the loss functions.
Revised: Figure 2 shows the relationship of the loss functions.

Original: We call our framework NRE - C 1 and expound the details in Section 2.
Revised: We call our framework NRE - C 1 and expound the details in Section 2.

Original: An existing importance sampling diagnostic [30] tests whether a classifier can distinguish p ( x | θ ) samples from from samples from p ( x ) weighted by the estimated ratio.
Revised: An existing importance sampling diagnostic [30] tests whether a classifier can distinguish p ( x | θ ) samples from from samples from p ( x ) weighted by the estimated ratio.

Original: We demonstrate that, when estimating accurate posteriors, our proposed NRE - C passes this diagnostic while NRE - B does not.
Revised: We demonstrate that, when estimating accurate posteriors, our proposed NRE - C passes this diagnostic while NRE - B does not.

Original: Taking inspiration from mutual information estimation [58], we propose applying a variational bound on the mutual information between θ and x in a novel way–as an informative metric measuring a lower bound on the Kullback-Leibler divergence between surrogate posterior estimate p w ( θ | x ) and p ( θ | x ) , averaged over p ( x ) .
Revised: Taking inspiration from mutual information estimation [58], we propose applying a variational bound on the mutual information between θ and x in a novel way–as an informative metric measuring a lower bound on the Kullback-Leibler divergence between surrogate posterior estimate p w ( θ | x ) and p ( θ | x ) , averaged over p ( x ) .

Original: Unlike with two-sample testing methods commonly used in machine learning literature [44], our metric samples only from p ( θ , x ) , which is always available in SBI , and does not require samples from the intractable p ( θ | x ) .
Revised: Unlike with two-sample testing methods commonly used in machine learning literature [44], our metric samples only from p ( θ , x ) , which is always available in SBI , and does not require samples from the intractable p ( θ | x ) .

Original: Our metric is meaningful to scientists working on problems with intractable posteriors.
Revised: Our metric is meaningful to scientists working on problems with intractable posteriors.

Original: The technique requires estimating the partition function, which can be expensive.
Revised: The technique requires estimating the partition function, which can be expensive.

Original: We find the metric to be well correlated with results from two-sample tests.
Revised: We find the metric to be well correlated with results from two-sample tests.

Original: We evaluate NRE - B and NRE - C in a fair comparison in several training regimes in Section 3.
Revised: We evaluate NRE - B and NRE - C in a fair comparison in several training regimes in Section 3.

Original: We perform a hyperparameter search on three simulators with tractable likelihood by benchmarking the behavior when (a) jointly drawn pairs ( θ , x ) are unlimited or when jointly drawn pairs ( θ , x ) are fixed but we (b) can draw from the prior p ( θ ) without limit or (c) are restricted to the initial pairs.
Revised: We perform a hyperparameter search on three simulators with tractable likelihood by benchmarking the behavior when (a) jointly drawn pairs ( θ , x ) are unlimited or when jointly drawn pairs ( θ , x ) are fixed but we (b) can draw from the prior p ( θ ) without limit or (c) are restricted to the initial pairs.

Original: We also perform the SBI benchmark of Lueckmann et al.
Revised: We also perform the SBI benchmark of Lueckmann et al.

Original: [44] with our recommended hyperparameters.
Revised: [44] with our recommended hyperparameters.

Original:  Methods
Revised:  Methods

Original: The ratio between probability distributions can be estimated using the “likelihood ratio trick” by training a classifier to distinguish samples [12, 19, 27, 30, 50, 66, 69].
Revised: The ratio between probability distributions can be estimated using the “likelihood ratio trick” by training a classifier to distinguish samples [12, 19, 27, 30, 50, 66, 69].

Original: We first summarize the loss functions of NRE - A and NRE - B which approximate the intractable likelihood-to-evidence ratio r ( x | θ ) :
Revised: We first summarize the loss functions of NRE - A and NRE - B which approximate the intractable likelihood-to-evidence ratio r ( x | θ ) :

Original: = p ( x | θ ) p ( x ) .
Revised: = p ( x | θ ) p ( x ) .

Original: We then elaborate on our proposed generalization, NRE - C .
Revised: We then elaborate on our proposed generalization, NRE - C .

Original: Finally, we explain how to recover NRE - A and NRE - B within our framework and comment on the normalization properties.
Revised: Finally, we explain how to recover NRE - A and NRE - B within our framework and comment on the normalization properties.

Original: NRE - A
Revised: NRE - A

Original: Hermans et al.
Revised: Hermans et al.

Original: [30] train a binary classifier to distinguish ( θ , x ) pairs drawn dependently p ( θ , x ) from those drawn independently p ( θ ) p ( x ) .
Revised: [30] train a binary classifier to distinguish ( θ , x ) pairs drawn dependently p ( θ , x ) from those drawn independently p ( θ ) p ( x ) .

Original: This classifier is parameterized by a neural network f w which approximates log r ( x | θ ) .
Revised: This classifier is parameterized by a neural network f w which approximates log r ( x | θ ) .

Original: We seek optimal network weights w ∗ ∈ arg min
Revised: We seek optimal network weights w ∗ ∈ arg min

Original: w
Revised: w

Original: [Equation]
Revised: [Equation]

Original: θ ( b ) , x ( b ) ∼ p ( θ ) p ( x ) and θ ( b ′ ) , x ( b ′ ) ∼ p
Revised: θ ( b ) , x ( b ) ∼ p ( θ ) p ( x ) and θ ( b ′ ) , x ( b ′ ) ∼ p

Original: θ , x ) over B samples.
Revised: θ , x ) over B samples.

Original: NRE - A ’s ratio estimate converges to f w ∗ = log p ( x | θ ) p ( x ) given unlimited model flexibility and data.
Revised: NRE - A ’s ratio estimate converges to f w ∗ = log p ( x | θ ) p ( x ) given unlimited model flexibility and data.

Original: Details can be found in Appendix A.
Revised: Details can be found in Appendix A.

Original: NRE - B
Revised: NRE - B

Original: Durkan et al.
Revised: Durkan et al.

Original: [16] train a classifier that selects from among K parameters ( θ 1 , . .
Revised: [16] train a classifier that selects from among K parameters ( θ 1 , . .

Original: . , θ K ) which could have generated x , in contrast with NRE - A ’s binary possibilities.
Revised: . , θ K ) which could have generated x , in contrast with NRE - A ’s binary possibilities.

Original: One of these parameters θ k is always drawn jointly with x .
Revised: One of these parameters θ k is always drawn jointly with x .

Original: The classifier is parameterized by a neural network g w which approximates log r ( x | θ ) .
Revised: The classifier is parameterized by a neural network g w which approximates log r ( x | θ ) .

Original: Training is done over B samples by finding
Revised: Training is done over B samples by finding

Original: [Equation]
Revised: [Equation]

Original: where θ ( b ′ ) 1 , . .
Revised: where θ ( b ′ ) 1 , . .

Original: , θ ( b ′ )
Revised: , θ ( b ′ )

Original: K ∼ p
Revised: K ∼ p

Original: ( θ ) and x ( b ′ )
Revised: ( θ ) and x ( b ′ )

Original: ∼ p ( x | θ ( b ′ ) k ) .
Revised: ∼ p ( x | θ ( b ′ ) k ) .

Original: Given unlimited model flexibility and data
Revised: Given unlimited model flexibility and data

Original: NRE - B ’s ratio estimate converges to g w ∗ ( θ , x )
Revised: NRE - B ’s ratio estimate converges to g w ∗ ( θ , x )

Original: = log p ( θ | x ) p ( θ ) + c w ∗ ( x ) .
Revised: = log p ( θ | x ) p ( θ ) + c w ∗ ( x ) .

Original: Details are in Appendix A.
Revised: Details are in Appendix A.

Original: 2.1 Contrastive Neural Ratio Estimation
Revised: 2.1 Contrastive Neural Ratio Estimation

Original: Our proposed algorithm NRE - C trains a classifier to identify which θ among K candidates is responsible for generating a given x , inspired by NRE - B .
Revised: Our proposed algorithm NRE - C trains a classifier to identify which θ among K candidates is responsible for generating a given x , inspired by NRE - B .

Original: We added another option that indicates x was drawn independently, inspired by NRE - A .
Revised: We added another option that indicates x was drawn independently, inspired by NRE - A .

Original: The introduction of the additional class yields a ratio without the specific c w ( x ) bias at optimum.
Revised: The introduction of the additional class yields a ratio without the specific c w ( x ) bias at optimum.

Original: Define Θ :=
Revised: Define Θ :=

Original: ( θ 1 , ..., θ K ) and conditional probability
Revised: ( θ 1 , ..., θ K ) and conditional probability

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: We set marginal probabilities p ( y = k ) :
Revised: We set marginal probabilities p ( y = k ) :

Original: = p K for all k ≥ 1 and p ( y = 0) :
Revised: = p K for all k ≥ 1 and p ( y = 0) :

Original: = p 0 , yielding the relationship p 

Revised: = p 0 , yielding the relationship p

Original: = 1 − Kp K .
Revised: = 1 − Kp K .

Original: Let the odds of any pair being drawn dependently to completely independently be γ :=
Revised: Let the odds of any pair being drawn dependently to completely independently be γ :=

Original: Kp K p 0 .
Revised: Kp K p 0 .

Original: We now use Bayes’ formula to compute the conditional probability p ( y
Revised: We now use Bayes’ formula to compute the conditional probability p ( y

Original: [Equation]
Revised: [Equation]

Original: We dropped the NRE - C subscript and substituted in γ to replace the p ( y ) class probabilities.
Revised: We dropped the NRE - C subscript and substituted in γ to replace the p ( y ) class probabilities.

Original: We train a classifier, parameterized by neural network h w ( θ , x ) with weights w , to approximate (6) by
Revised: We train a classifier, parameterized by neural network h w ( θ , x ) with weights w , to approximate (6) by

Original: [Equation]
Revised: [Equation]

Original: ( y =
Revised: ( y =

Original: [Equation]
Revised: [Equation]

Original: |
Revised: |

Original: [Equation]
Revised: [Equation]

Original: , x
Revised: , x

Original: [Equation]
Revised: [Equation]

Original: We note that (7) still satisfies (cid:80) Kk =

Revised: We note that (7) still satisfies (cid:80) Kk =

Original: q w ( y = k
Revised: q w ( y = k

Original: | Θ , x ) = 1 , no matter the parameterization.
Revised: | Θ , x ) = 1 , no matter the parameterization.

Original: Optimization We design a loss function that encourages h w ( θ , x )
Revised: Optimization We design a loss function that encourages h w ( θ , x )

Original: = log p ( x | θ ) p ( x ) at convergence, and holds at optimum with unlimited flexibility and data.
Revised: = log p ( x | θ ) p ( x ) at convergence, and holds at optimum with unlimited flexibility and data.

Original: We introduce the cross entropy loss
Revised: We introduce the cross entropy loss

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: and minimize it towards w ∗ ∈ arg min w ℓ ( w ) .
Revised: and minimize it towards w ∗ ∈ arg min w ℓ ( w ) .

Original: We point out that the final term is symmetric up to permutation of Θ , enabling the replacement of the sum by multiplication with K .
Revised: We point out that the final term is symmetric up to permutation of Θ , enabling the replacement of the sum by multiplication with K .

Original: When γ and K are known, p 

Revised: When γ and K are known, p

Original: = 11+ γ and p K = 1 K γ 1+ γ under our constraints.
Revised: = 11+ γ and p K = 1 K γ 1+ γ under our constraints.

Original: Without loss of generality, we let θ 1 , . . .
Revised: Without loss of generality, we let θ 1 , . . .

Original: , θ K ∼ p ( θ ) and x ∼ p ( x | θ K ) .
Revised: , θ K ∼ p ( θ ) and x ∼ p ( x | θ K ) .

Original: An empirical estimate of the loss on B samples is therefore
Revised: An empirical estimate of the loss on B samples is therefore

Original: [Equation]
Revised: [Equation]

Original: In the first term, the classifier sees a completely independently drawn sample of x and Θ while θ K is drawn jointly with x in the second term.
Revised: In the first term, the classifier sees a completely independently drawn sample of x and Θ while θ K is drawn jointly with x in the second term.

Original: In both terms, the classifier considers K choices.
Revised: In both terms, the classifier considers K choices.

Original: In practice, we bootstrap both θ ( b ) 1 , . . .
Revised: In practice, we bootstrap both θ ( b ) 1 , . . .

Original: , θ ( b ) K and θ ( b ′ ) 1 , . . .
Revised: , θ ( b ) K and θ ( b ′ ) 1 , . . .

Original: , θ ( b ′ )
Revised: , θ ( b ′ )

Original: K − 1 from the same mini-batch and compare them to the same x , similarly to NRE - A and NRE - B .
Revised: K − 1 from the same mini-batch and compare them to the same x , similarly to NRE - A and NRE - B .

Original: Proof of the above is in Appendix B.
Revised: Proof of the above is in Appendix B.

Original: Recovering NRE - A and NRE - B
Revised: Recovering NRE - A and NRE - B

Original: NRE - C is general because specific hyperparameter settings recover
Revised: NRE - C is general because specific hyperparameter settings recover

Original: NRE - A and NRE - B .
Revised: NRE - A and NRE - B .

Original: To recover NRE - A one should set γ = 1 and K = 1 in (9) yielding
Revised: To recover NRE - A one should set γ = 1 and K = 1 in (9) yielding

Original: [Equation]
Revised: [Equation]

Original: where we dropped the lower index.
Revised: where we dropped the lower index.

Original: Recovering NRE - B requires taking the limit γ → ∞ in the loss function.
Revised: Recovering NRE - B requires taking the limit γ → ∞ in the loss function.

Original: In that case, the first term goes to zero, and second term converges to the softmax function.
Revised: In that case, the first term goes to zero, and second term converges to the softmax function.

Original: [Equation]
Revised: [Equation]

Original: is determined by substitution into (9).
Revised: is determined by substitution into (9).

Original: Both equations are obviously the same as their counterparts.
Revised: Both equations are obviously the same as their counterparts.

Original: Estimating a normalized posterior In the limit of infinite data and infinite neural network capacity (width, depth) the optimal classifier trained using NRE - C (with γ ∈ R + ) satisfies the equality:
Revised: Estimating a normalized posterior In the limit of infinite data and infinite neural network capacity (width, depth) the optimal classifier trained using NRE - C (with γ ∈ R + ) satisfies the equality:

Original: [Equation]
Revised: [Equation]

Original: In particular, we have that the following normalizing constant is trivial:
Revised: In particular, we have that the following normalizing constant is trivial:

Original: [Equation]
Revised: [Equation]

Original: This is a result of Lemma 1 in Appendix B.
Revised: This is a result of Lemma 1 in Appendix B.

Original: However, practitioners never operate in this setting, rather they use finite sample sizes and neural networks with limited capacity that are optimized locally.
Revised: However, practitioners never operate in this setting, rather they use finite sample sizes and neural networks with limited capacity that are optimized locally.

Original: The non-optimal function exp( h w ( θ , x )) does not have a direct interpretation as a ratio of probability distributions, rather as the function to weigh the prior p ( θ ) to approximate the unnormalized posterior.
Revised: The non-optimal function exp( h w ( θ , x )) does not have a direct interpretation as a ratio of probability distributions, rather as the function to weigh the prior p ( θ ) to approximate the unnormalized posterior.

Original: In other words, we find the following approximation for the posterior p ( θ | x ) :
Revised: In other words, we find the following approximation for the posterior p ( θ | x ) :

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where in general the normalizing constant is not trivial, i.e. Z w ( x ) ̸ = 1 .
Revised: where in general the normalizing constant is not trivial, i.e. Z w ( x ) ̸ = 1 .

Original: As stated above, the NRE - C (and NRE - A ) objective encourages Z w ( x ) to converge to 1 .
Revised: As stated above, the NRE - C (and NRE - A ) objective encourages Z w ( x ) to converge to 1 .

Original: This is in sharp contrast to NRE - B , where even at optimum with an unrestricted function class a non-trivial x -dependent bias term can appear.
Revised: This is in sharp contrast to NRE - B , where even at optimum with an unrestricted function class a non-trivial x -dependent bias term can appear.

Original: There is no restriction on how pathological the NRE - B bias c w ( x ) can be.
Revised: There is no restriction on how pathological the NRE - B bias c w ( x ) can be.

Original: Consider a minimizer of (4), the NRE - B loss function, h w ∗ + c w ∗ ( x ) .
Revised: Consider a minimizer of (4), the NRE - B loss function, h w ∗ + c w ∗ ( x ) .

Original: Adding any function d ( x ) cancels out in the fraction and is also a minimizer of (4).
Revised: Adding any function d ( x ) cancels out in the fraction and is also a minimizer of (4).

Original: This freedom complicates any numerical computation of the normalizing constant and renders the importance sampling diagnostic from Section 2.2 generally inapplicable.
Revised: This freedom complicates any numerical computation of the normalizing constant and renders the importance sampling diagnostic from Section 2.2 generally inapplicable.

Original: We report Monte Carlo estimates of Z w ( x ) on a test problem across hyperparameters in Figure 14.
Revised: We report Monte Carlo estimates of Z w ( x ) on a test problem across hyperparameters in Figure 14.

Original: 2.2 Measuring performance & ratio estimator diagnostics
Revised: 2.2 Measuring performance & ratio estimator diagnostics

Original: SBI is difficult to verify because, for many use cases, the practitioner cannot compare surrogate p w ( θ | x ) to the intractable ground truth p
Revised: SBI is difficult to verify because, for many use cases, the practitioner cannot compare surrogate p w ( θ | x ) to the intractable ground truth p

Original: ( θ | x ) .
Revised: ( θ | x ) .

Original: Incongruous with the practical use case for SBI , much of the literature has focused on measuring the similarity between surrogate and posterior using two-samples tests on tractable problems.
Revised: Incongruous with the practical use case for SBI , much of the literature has focused on measuring the similarity between surrogate and posterior using two-samples tests on tractable problems.

Original: For comparison with literature, we first reference a two-sample exactness metric which requires a tractable posterior.
Revised: For comparison with literature, we first reference a two-sample exactness metric which requires a tractable posterior.

Original: We then discuss diagnostics which do not require samples from p ( θ | x ) , commenting on the relevance for each NRE algorithm with empirical results.
Revised: We then discuss diagnostics which do not require samples from p ( θ | x ) , commenting on the relevance for each NRE algorithm with empirical results.

Original: Further, we find that a known variational bound to the mutual information is tractable to estimate within SBI , that it bounds the average Kullback-Leibler divergence between surrogate and posterior, and propose to use it for model comparison on intractable inference tasks.
Revised: Further, we find that a known variational bound to the mutual information is tractable to estimate within SBI , that it bounds the average Kullback-Leibler divergence between surrogate and posterior, and propose to use it for model comparison on intractable inference tasks.

Original: Comparing to a tractable posterior with estimates of exactness Assessments of approximate posterior quality are available when samples can be drawn from both the posterior θ ∼ p ( θ |
Revised: Comparing to a tractable posterior with estimates of exactness Assessments of approximate posterior quality are available when samples can be drawn from both the posterior θ ∼ p ( θ |

Original: x ) and the approximation θ ∼ q ( θ | x ) .
Revised: x ) and the approximation θ ∼ q ( θ | x ) .

Original: In the deep learning-based SBI literature, exactness is measured as a function of computational cost, usually simulator calls.
Revised: In the deep learning-based SBI literature, exactness is measured as a function of computational cost, usually simulator calls.

Original: We investigate this with NRE - C in Section 3.3.
Revised: We investigate this with NRE - C in Section 3.3.

Original: Based on the recommendations of Lueckmann et al.
Revised: Based on the recommendations of Lueckmann et al.

Original: [44] our experimental results are measured using the Classifier Two-Sample Test (C2ST) [17, 40, 41].
Revised: [44] our experimental results are measured using the Classifier Two-Sample Test (C2ST) [17, 40, 41].

Original: A classifier is trained to distinguish samples from either the surrogate or the ground truth posterior.
Revised: A classifier is trained to distinguish samples from either the surrogate or the ground truth posterior.

Original: An average classification probability on holdout data of 1.0 implies that samples from each distribution are easily identified; 0.5 implies either the distributions are the same or the classifier does not have the capacity to distinguish them.
Revised: An average classification probability on holdout data of 1.0 implies that samples from each distribution are easily identified; 0.5 implies either the distributions are the same or the classifier does not have the capacity to distinguish them.

Original: [Figure]
Revised: [Figure]

Original: P o s t
Revised: P o s t

Original: e r
Revised: e r

Original: i o r d e n s i t y
Revised: i o r d e n s i t y

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: r o x i m
Revised: r o x i m

Original: [Figure]
Revised: [Figure]

Original: t
Revised: t

Original: [Figure]
Revised: [Figure]

Original: li k e
Revised: li k e

Original: [Figure]
Revised: [Figure]

Original: d d e n s i t y ,
Revised: d d e n s i t y ,

Original: [Figure]
Revised: [Figure]

Original: a r i
Revised: a r i

Original: [Figure]
Revised: [Figure]

Original: u s
Revised: u s

Original: [Figure]
Revised: [Figure]

Original: Importance sampling diagnostic
Revised: Importance sampling diagnostic

Original: An accurate likelihood-to-evidence weight transforms the data distribution into the likelihood by p ( x | θ )
Revised: An accurate likelihood-to-evidence weight transforms the data distribution into the likelihood by p ( x | θ )

Original: =
Revised: =

Original: p ( x ) r ( x | θ ) .
Revised: p ( x ) r ( x | θ ) .

Original: Since NRE necessitates simulator access, we can test the ratio estimator by training a classifier to distinguish unweighted p ( x | θ ) samples from weighted p ( x )ˆ r ( x | θ ) samples, where ˆ r implies an estimate.
Revised: Since NRE necessitates simulator access, we can test the ratio estimator by training a classifier to distinguish unweighted p ( x | θ ) samples from weighted p ( x )ˆ r ( x | θ ) samples, where ˆ r implies an estimate.

Original: Indistinguishability between samples implies either that the approximate ratio is accurate for parameter θ or that the classifier does not have sufficient power to find predictive features.
Revised: Indistinguishability between samples implies either that the approximate ratio is accurate for parameter θ or that the classifier does not have sufficient power to find predictive features.

Original: Issues with classification power can be detected by assessing the classifier’s ability to distinguish p ( x ) from p ( x | θ ) .
Revised: Issues with classification power can be detected by assessing the classifier’s ability to distinguish p ( x ) from p ( x | θ ) .

Original: The performance can be visualized in a receiver operating curve ( ROC ) or measured by the area under the curve ( ROC AUC ).
Revised: The performance can be visualized in a receiver operating curve ( ROC ) or measured by the area under the curve ( ROC AUC ).

Original: This diagnostic has been used for ratio estimators before [12, 30] but it comes from training models under covariate shift [62].
Revised: This diagnostic has been used for ratio estimators before [12, 30] but it comes from training models under covariate shift [62].

Original: It is particularly appealing because it does not require samples from p ( θ | x ) .
Revised: It is particularly appealing because it does not require samples from p ( θ | x ) .

Original: Durkan et al.
Revised: Durkan et al.

Original: [16] do not mention this diagnostic in their paper, but due to its intrinsic bias NRE - B does not fulfill the identity necessary for this diagnostic to hold at optimum.
Revised: [16] do not mention this diagnostic in their paper, but due to its intrinsic bias NRE - B does not fulfill the identity necessary for this diagnostic to hold at optimum.

Original: The unknown factor that depends on x implies p ( x | θ ) ̸ = p ( x ) exp ◦ g w ( x | θ ) .
Revised: The unknown factor that depends on x implies p ( x | θ ) ̸ = p ( x ) exp ◦ g w ( x | θ ) .

Original: We provide empirical evidence of this issue in Figure 3.
Revised: We provide empirical evidence of this issue in Figure 3.

Original: Although NRE - B accurately approximates the true posterior, it demonstrably fails the diagnostic.
Revised: Although NRE - B accurately approximates the true posterior, it demonstrably fails the diagnostic.

Original: Given the limited options for verification of SBI results, this presents a major problem by significantly limiting the trustworthiness of NRE - B on any problem with an intractable posterior.
Revised: Given the limited options for verification of SBI results, this presents a major problem by significantly limiting the trustworthiness of NRE - B on any problem with an intractable posterior.

Original: In
Revised: In

Original: Appendix B, we show that the unrestricted NRE - B -specific c w ( x ) bias means approximating p ( x | θ ) with normalized importance weights will not solve the issue.
Revised: Appendix B, we show that the unrestricted NRE - B -specific c w ( x ) bias means approximating p ( x | θ ) with normalized importance weights will not solve the issue.

Original: Mutual information bound Selecting the surrogate model most-similar to the target posterior remains intractable without access to p ( θ | x o ) .
Revised: Mutual information bound Selecting the surrogate model most-similar to the target posterior remains intractable without access to p ( θ | x o ) .

Original: Nevertheless, practitioners must decide which surrogate should approximate the posterior across training and hyperparameter search.
Revised: Nevertheless, practitioners must decide which surrogate should approximate the posterior across training and hyperparameter search.

Original: Unfortunately, the validation losses between different versions of NRE and different K and γ settings are not comparable.
Revised: Unfortunately, the validation losses between different versions of NRE and different K and γ settings are not comparable.

Original: A good heuristic is to choose the model which minimizes the Kullback-Leibler divergence on average over possible data p ( x ) .
Revised: A good heuristic is to choose the model which minimizes the Kullback-Leibler divergence on average over possible data p ( x ) .

Original: In Appendix D, we prove the relationship between I ( θ ; x ) , the mutual information with respect to p ( θ , x ) , our models’ variational bound I (0) w ( θ ; x ) , and the average KLD
Revised: In Appendix D, we prove the relationship between I ( θ ; x ) , the mutual information with respect to p ( θ , x ) , our models’ variational bound I (0) w ( θ ; x ) , and the average KLD

Original: [Equation]
Revised: [Equation]

Original: The non-negativity of all terms in (15) implies I ( θ ; x ) ≥ I (0) w ( θ ; x ) ; that means the model which minimizes − I (0) w ( θ ; x ) best satisfies our heuristic.
Revised: The non-negativity of all terms in (15) implies I ( θ ; x ) ≥ I (0) w ( θ ; x ) ; that means the model which minimizes − I (0) w ( θ ; x ) best satisfies our heuristic.

Original: We propose to approximate − I (0) w ( θ ; x ) with
Revised: We propose to approximate − I (0) w ( θ ; x ) with

Original: Monte Carlo using held-out data as a metric for model selection during training and across hyperparameters.
Revised: Monte Carlo using held-out data as a metric for model selection during training and across hyperparameters.

Original: The expectation values average over p ( θ , x ) , p ( θ ) , and p ( x ) .
Revised: The expectation values average over p ( θ , x ) , p ( θ ) , and p ( x ) .

Original: We can sample from all of these distributions in the SBI context.
Revised: We can sample from all of these distributions in the SBI context.

Original: Since the second term in (16) computes the average log partition function, our metric can compare NRE - B -based surrogates to NRE - C -based ones.
Revised: Since the second term in (16) computes the average log partition function, our metric can compare NRE - B -based surrogates to NRE - C -based ones.

Original: However, the metric comes with the normal challenges of estimating the log partition function which can be very expensive or high variance in some cases.
Revised: However, the metric comes with the normal challenges of estimating the log partition function, which can be very expensive.

Revised: Additionally, the presence of the ratio in the integrand can make this integral high variance.

Original: We  go into  more depth in Appendix D, including mentioning the relevance to Neural Posterior Estimation [16, 21, 42, 53].
Revised: We treat a generally tractable bound in Appendix D. We go into much more depth there and discuss of the relevance to Neural Posterior Estimation [16, 21, 42, 53].

Original: While the application to SBI is novel, bounds on the mutual information have been broadly investigated for contrastive representation learning and mutual information estimation [4, 22, 23, 25, 58, 71].
Revised: While the application to SBI is novel, bounds on the mutual information have been broadly investigated for contrastive representation learning and mutual information estimation [4, 22, 23, 25, 58, 71].

Original: Empirical expected coverage probability For a candidate distribution to qualify as the posterior, integrating over data must return the prior.
Revised: Empirical expected coverage probability For a candidate distribution to qualify as the posterior, integrating over data must return the prior.

Original: A measurement that follows from calibration to the prior is called expected coverage probability.
Revised: A measurement that follows from calibration to the prior is called expected coverage probability.

Original: Expected coverage probability can be estimated with samples from p ( θ , x ) and any amortized SBI method.
Revised: Expected coverage probability can be estimated with samples from p ( θ , x ) and any amortized SBI method.

Original: Although important, ability to compute this metric does not distinguish NRE - C .
Revised: Although important, ability to compute this metric does not distinguish NRE - C .

Original: We refer the interested reader to Hermans et al.
Revised: We refer the interested reader to Hermans et al.

Original: We note that popular sequential techniques generally render this diagnostic inapplicable, with exceptions [10, 47, 48].
Revised: We note that popular sequential techniques generally render this diagnostic inapplicable, with exceptions [10, 47, 48].

Original:  Experiments
Revised:  Experiments

Original: We perform experiments in three settings to measure the exactness of surrogate models under various hyperparameters and training regimes.
Revised: We perform experiments in three settings to measure the exactness of surrogate models under various hyperparameters and training regimes.

Original: Section 3.1 aims to identify whether data or architecture is the bottleneck in accuracy by drawing from the joint with every mini-batch.
Revised: Section 3.1 aims to identify whether data or architecture is the bottleneck in accuracy by drawing from the joint with every mini-batch.

Original: The next experiments aim to determine how to optimally extract inference information given a limited amount of simulation data.
Revised: The next experiments aim to determine how to optimally extract inference information given a limited amount of simulation data.

Original: Section 3.2 leverages a cheap prior by drawing new contrastive parameters with every mini-batch.
Revised: Section 3.2 leverages a cheap prior by drawing new contrastive parameters with every mini-batch.

Original: Finally, Section 3.3 applies the commonplace training regime for deep learning-based SBI literature of fixed data and bootstrapped contrastive parameters.
Revised: Finally, Section 3.3 applies the commonplace training regime for deep learning-based SBI literature of fixed data and bootstrapped contrastive parameters.

Original: In this setting, we also benchmark NRE - C on ten inference tasks from Lueckmann et al.
Revised: In this setting, we also benchmark NRE - C on ten inference tasks from Lueckmann et al.

Original: [44] using our recommended hyperparameters.
Revised: [44] using our recommended hyperparameters.

Original: On all hyperparameter searches we consider three simulators from the simulation-based inference benchmark, namely SLCP, Two Moons, and Gaussian Mixture [44].
Revised: On all hyperparameter searches we consider three simulators from the simulation-based inference benchmark, namely SLCP, Two Moons, and Gaussian Mixture [44].

Original: SLCP is a difficult inference task which has a simple likelihood and complex posterior [21, 56].
Revised: SLCP is a difficult inference task which has a simple likelihood and complex posterior [21, 56].

Original: Parameters are five dimensional and the data are four samples from a two dimensional random variable.
Revised: Parameters are five dimensional and the data are four samples from a two dimensional random variable.

Original: Two Moons is two dimensional with a crescent-shaped bimodal posterior.
Revised: Two Moons is two dimensional with a crescent-shaped bimodal posterior.

Original: Finally, the Gaussian Mixture data draws from a mixture of two, two-dimensional normal distributions with extremely different covariances [3, 63, 64, 70].
Revised: Finally, the Gaussian Mixture data draws from a mixture of two, two-dimensional normal distributions with extremely different covariances [3, 63, 64, 70].

Original: Our surrogate models are parameterized by one of these architectures: Small NN is like the benchmark with 50 hidden units and two residual blocks.
Revised: Our surrogate models are parameterized by one of these architectures: Small NN is like the benchmark with 50 hidden units and two residual blocks.

Original: Large NN has 128 hidden units and three residual blocks.
Revised: Large NN has 128 hidden units and three residual blocks.

Original: We use batch normalization, unlike the benchmark.
Revised: We use batch normalization, unlike the benchmark.

Original: We compare their performance on a grid of γ and K values.
Revised: We compare their performance on a grid of γ and K values.

Original: We report post-training results using the C2ST, and mid-training validation loss for insight into convergence rate.
Revised: We report post-training results using the C2ST, and mid-training validation loss for insight into convergence rate.

Original: We generally use residual networks [28] with batch normalization [33] and train them using adam [37].
Revised: We generally use residual networks [28] with batch normalization [33] and train them using adam [37].

Original: We also run NRE - B with the same architecture for comparison.
Revised: We also run NRE - B with the same architecture for comparison.

Original: To compare with NRE - B we set the number of total contrastive parameters equal.
Revised: To compare with NRE - B we set the number of total contrastive parameters equal.

Original: What does fair comparison mean in our experimentation?
Revised: What does fair comparison mean in our experimentation?

Original: We compare models across fixed number of gradient steps.
Revised: We compare models across fixed number of gradient steps.

Original: This implies that models with more classes, i.e., greater K , evaluate the classifierin-training on more pairs at a given training step than a model with fewer classes.
Revised: This implies that models with more classes, i.e., greater K , evaluate the classifierin-training on more pairs at a given training step than a model with fewer classes.

Original: An alternative which we do not apply in this paper: Vary the number of gradient steps, holding the number of pair evaluations fixed, i.e., a model with higher K sees the same number of pairs as a model with lower
Revised: An alternative which we do not apply in this paper: Vary the number of gradient steps, holding the number of pair evaluations fixed, i.e., a model with higher K sees the same number of pairs as a model with lower

Original: K but the model with lower K has been updated more times.
Revised: K but the model with lower K has been updated more times.

Original: We leave this analysis for future work.
Revised: We leave this analysis for future work.

Original: 3.1 Behavior with unlimited data
Revised: 3.1 Behavior with unlimited data

Original: What is responsible for inaccuracies of the surrogate model when training has saturated?
Revised: What is responsible for inaccuracies of the surrogate model when training has saturated?

Original: (a) the amount of training data (b) the flexibility of the model?
Revised: (a) the amount of training data (b) the flexibility of the model?

Original: In this section we provide new simulation and parameter data with every mini-batch and train until saturation, thereby eliminating the possibility of (a).
Revised: In this section we provide new simulation and parameter data with every mini-batch and train until saturation, thereby eliminating the possibility of (a).

Original: The newly drawn mini-batch parameters Θ are bootstrapped for the contrastive pairs.
Revised: The newly drawn mini-batch parameters Θ are bootstrapped for the contrastive pairs.

Original: The setting is similar to REJ - ABC where simulations are drawn until the posterior has converged.
Revised: The setting is similar to REJ - ABC where simulations are drawn until the posterior has converged.

Original: The results of this study will provide a baseline to compare with limited-data results and help us understand how the deep learning architecture’s limitations are affected by our introduced hyperparameters.
Revised: The results of this study will provide a baseline to compare with limited-data results and help us understand how the deep learning architecture’s limitations are affected by our introduced hyperparameters.

Original: The results are reported in the top row of Figure 5.
Revised: The results are reported in the top row of Figure 5.

Original: The trend is that increasing the number of contrastive examples helps NRE - B and NRE - C .
Revised: The trend is that increasing the number of contrastive examples helps NRE - B and NRE - C .

Original: There is not a clear trend with respect to γ in the C2ST.
Revised: There is not a clear trend with respect to γ in the C2ST.

Original: Study of the detailed validation losses in Appendix C reveals that high γ is associated with higher variance validation loss during training, but here it does not seem to strongly affect convergence rate.
Revised: Study of the detailed validation losses in Appendix C reveals that high γ is associated with higher variance validation loss during training, but here it does not seem to strongly affect convergence rate.

Original: Saturation is reached before the maximum number of epochs have elapsed and Large NN converges to a better result.
Revised: Saturation is reached before the maximum number of epochs have elapsed and Large NN converges to a better result.

Original: Although obfuscated in the averaged results in Figure 5, the result is obvious on SLCP, the most difficult inference task that provides an opportunity for a more flexible architecture to improve.
Revised: Although obfuscated in the averaged results in Figure 5, the result is obvious on SLCP, the most difficult inference task that provides an opportunity for a more flexible architecture to improve.

Original: We argue that generally the performance bottleneck has to do with the number of contrastive parameters, network flexibility, or training details rather than the amount of data.
Revised: We argue that generally the performance bottleneck has to do with the number of contrastive parameters, network flexibility, or training details rather than the amount of data.

Original: Despite saturating validation losses, the C2ST improves based on these factors.
Revised: Despite saturating validation losses, the C2ST improves based on these factors.

Original: Intuitively a more flexible model could continue to extract information from this unlimited set of jointly drawn ( θ , x ) pairs.
Revised: Intuitively a more flexible model could continue to extract information from this unlimited set of jointly drawn ( θ , x ) pairs.

Original: This has some consequence because network flexibility may limit performance in the benchmark case.
Revised: This has some consequence because network flexibility may limit performance in the benchmark case.

Original: Appendix C contains more detailed results.
Revised: Appendix C contains more detailed results.

Original: 3.2 Leveraging fast priors (drawing theta)
Revised: 3.2 Leveraging fast priors (drawing theta)

Original: In practice, the simulator is often slow but one can often draw from the prior at the same pace as training a neural network for one mini-batch.
Revised: In practice, the simulator is often slow but one can often draw from the prior at the same pace as training a neural network for one mini-batch.

Original: Our goal is to understand how our hyperparameter are affected by this setting and whether it is valuable to use this technique in practice.
Revised: Our goal is to understand how our hyperparameter are affected by this setting and whether it is valuable to use this technique in practice.

Original: To our knowledge, this training regime has not been explored in the deep learning-based SBI literature.
Revised: To our knowledge, this training regime has not been explored in the deep learning-based SBI literature.

Original: Initially we draw a fixed set of around 20,000 samples ( θ , x ) ∼ p
Revised: Initially we draw a fixed set of around 20,000 samples ( θ , x ) ∼ p

Original: θ , x ) .
Revised: θ , x ) .

Original: For every mini-batch during training, we sample all necessary contrastive parameters from the prior.
Revised: For every mini-batch during training, we sample all necessary contrastive parameters from the prior.

Original: For each term in ˆ ℓ γ,K we take the same batch of contrastive parameters and reshuffle them, thereby equalizing the number of samples seen by NRE - B and NRE - C , up to bootstrap.
Revised: For each term in ˆ ℓ γ,K we take the same batch of contrastive parameters and reshuffle them, thereby equalizing the number of samples seen by NRE - B and NRE - C , up to bootstrap.

Original: The averaged C2ST results from the three inference tasks are reported in the middle row of Figure 5.
Revised: The averaged C2ST results from the three inference tasks are reported in the middle row of Figure 5.

Original: The resulting estimators are markedly less sensitive to the number of contrastive parameters than in
Revised: The resulting estimators are markedly less sensitive to the number of contrastive parameters than in

Original: Section 3.1.
Revised: Section 3.1.

Original: The number of network parameters has a positive effect, although again this is best seen on the hardest task, SLCP, in the Appendix C. Empirically, lower values of γ improve performance with Large NN; however, there is still high variance and this result may be from noise.
Revised: The number of network parameters has a positive effect, although again this is best seen on the hardest task, SLCP, in the Appendix C. Empirically, lower values of γ improve performance with Large NN; however, there is still high variance and this result may be from noise.

Original: In this setting, γ ̸ = 1 has a very small negative effect on convergence rate compared to unity, as seen in the validation loss plots in Appendix C. Since training has saturated, we claim this result implies drawing contrastive parameters from the prior helps extract the maximum amount of information from fixed simulation data x –without being strongly affected by the number of contrastive parameters K .
Revised: In this setting, γ ̸ = 1 has a very small negative effect on convergence rate compared to unity, as seen in the validation loss plots in Appendix C. Since training has saturated, we claim this result implies drawing contrastive parameters from the prior helps extract the maximum amount of information from fixed simulation data x –without being strongly affected by the number of contrastive parameters K .

Original: [Figure]
Revised: [Figure]

Original: 3.3 Simulation-based inference benchmark
Revised: 3.3 Simulation-based inference benchmark

Original: In this section, we assume the traditional literature setting of limited simulation and prior budget.
Revised: In this section, we assume the traditional literature setting of limited simulation and prior budget.

Original: Once we’ve selected hyperparameters based on a grid search of a subset of the SBI benchmark, we perform the entire benchmark with those hyperparameters.
Revised: Once we’ve selected hyperparameters based on a grid search of a subset of the SBI benchmark, we perform the entire benchmark with those hyperparameters.

Original: [Figure]
Revised: [Figure]

Original: Hyperparameter search To compare to the previous two sections, the amount of training data was fixed such that each epoch was comparable to Section 3.1, this amounts to about 20,000 samples.
Revised: Hyperparameter search To compare to the previous two sections, the amount of training data was fixed such that each epoch was comparable to Section 3.1, this amounts to about 20,000 samples.

Original: We first inspect the C2ST results in the last row of Figure 5.
Revised: We first inspect the C2ST results in the last row of Figure 5.

Original: The sensitivity to K appears less than in Section 3.1 but more than in Section 3.2.
Revised: The sensitivity to K appears less than in Section 3.1 but more than in Section 3.2.

Original: Smaller γ slightly improves
Revised: Smaller γ slightly improves

Original: Large NN’s performance but the noise makes this result uncertain.
Revised: Large NN’s performance but the noise makes this result uncertain.

Original: The larger network performs better, see Appendix C.
Revised: The larger network performs better, see Appendix C.

Original: We do a Monte Carlo estimate of our proposed metric, denoted − ˆ I (0) w ( θ ; x ) , to show performance on the SLCP task as a function of epochs in Figure 4.
Revised: We do a Monte Carlo estimate of our proposed metric, denoted − ˆ I (0) w ( θ ; x ) , to show performance on the SLCP task as a function of epochs in Figure 4.

Original: For both NRE - B and NRE - C , increasing K tends to positively affect the convergence rate.
Revised: For both NRE - B and NRE - C , increasing K tends to positively affect the convergence rate.

Original: Generally, with fixed γ and high K , peak performance is negatively impacted, in contrast to results from Figure 5 using the C2ST.
Revised: Generally, with fixed γ and high K , peak performance is negatively impacted, in contrast to results from Figure 5 using the C2ST.

Original: Generally, with fixed K , small γ slows convergences but leads to better optima than large γ .
Revised: Generally, with fixed K , small γ slows convergences but leads to better optima than large γ .

Original: When γ is fixed, small K slows convergence but leads to better optima than high K .
Revised: When γ is fixed, small K slows convergence but leads to better optima than high K .

Original: See Figure 12.
Revised: See Figure 12.

Original: Our take-away is that γ ∈ [0 . 1 , 10] has a small effect, otherwise learning can become unstable or very slow.
Revised: Our take-away is that γ ∈ [0 . 1 , 10] has a small effect, otherwise learning can become unstable or very slow.

Original: γ = 1 has a good compromise on high convergence rate without sacrificing too much performance.
Revised: γ = 1 has a good compromise on high convergence rate without sacrificing too much performance.

Original: Generally, we saw improved performance on the C2ST by increasing contrastive parameters K .
Revised: Generally, we saw improved performance on the C2ST by increasing contrastive parameters K .

Original: However, larger K also increases the magnitude of the normalizing constant, and extreme values reduced the performance on − ˆ I (0) w ( θ ; x ) , see Appendix D. Since the benchmark compares C2ST, we optimized our architecture based on that metric.
Revised: However, larger K also increases the magnitude of the normalizing constant, and extreme values reduced the performance on − ˆ I (0) w ( θ ; x ) , see Appendix D. Since the benchmark compares C2ST, we optimized our architecture based on that metric.

Original: Due to bootstrapping parameters from the batched θ , the maximum K is B/ 2 without reusing any θ to compare with x .
Revised: Due to bootstrapping parameters from the batched θ , the maximum K is B/ 2 without reusing any θ to compare with x .

Original: This is considering both terms in our loss function.
Revised: This is considering both terms in our loss function.

Original: Benchmark We performed the benchmark tasks from Lueckmann et al.
Revised: Benchmark We performed the benchmark tasks from Lueckmann et al.

Original: [44] with NRE - C .
Revised: [44] with NRE - C .

Original: Our architecture and hyperparameters deviate based on the results of the last paragraph.
Revised: Our architecture and hyperparameters deviate based on the results of the last paragraph.

Original: We applied the largest number of computationally practical contrastive parameters, namely K = 99 , and set γ = 1 .
Revised: We applied the largest number of computationally practical contrastive parameters, namely K = 99 , and set γ = 1 .

Original: Since we identified potential architecture bottlenecks in Sections 3.1 and 3.3, we used Large NN for our results.
Revised: Since we identified potential architecture bottlenecks in Sections 3.1 and 3.3, we used Large NN for our results.

Original: The averaged results are visible in Table 1.
Revised: The averaged results are visible in Table 1.

Original: Sequential estimates are prepended with an S , all other estimates are amortized.
Revised: Sequential estimates are prepended with an S , all other estimates are amortized.

Original: We trained a five amortized estimators per task with five seeds, then we predicted the posterior for each of the ten observations x 1 , . . .
Revised: We trained a five amortized estimators per task with five seeds, then we predicted the posterior for each of the ten observations x 1 , . . .

Original: , x 10 .
Revised: , x 10 .

Original: Lueckmann et al.
Revised: Lueckmann et al.

Original: [44] trained an estimator for every x with a new seed.
Revised: [44] trained an estimator for every x with a new seed.

Original: The reported C2ST for our method is averaged across ten posteriors per seed with five seeds, i.e., fifty C2ST computations per task.
Revised: The reported C2ST for our method is averaged across ten posteriors per seed with five seeds, i.e., fifty C2ST computations per task.

Original: The other methods are averaged across one posterior per seed with ten seeds, i.e., ten C2ST computations per task.
Revised: The other methods are averaged across one posterior per seed with ten seeds, i.e., ten C2ST computations per task.

Original: NRE - C performed better than NRE - B at all budgets and generally well among the amortized algorithms.
Revised: NRE - C performed better than NRE - B at all budgets and generally well among the amortized algorithms.

Original: The high performance at 10 5 simulation budget implies that our design has high capacity and may scale well.
Revised: The high performance at 10 5 simulation budget implies that our design has high capacity and may scale well.

Original: Sequential methods naturally lead to high exactness since they are tailored to the specific observation x o , but they come with drawbacks since they cannot practically perform empirical expected coverage tests [31, 48] nor bound the mutual information, as discussed in Section 2.2.
Revised: Sequential methods naturally lead to high exactness since they are tailored to the specific observation x o , but they come with drawbacks since they cannot practically perform empirical expected coverage tests [31, 48] nor bound the mutual information, as discussed in Section 2.2.

Original: Detailed results per task can be found in Appendix C.
Revised: Detailed results per task can be found in Appendix C.

Original:  Conclusion
Revised:  Conclusion

Original: [Table]
Revised: [Table]

Original: We introduced a generalization of NRE - A and
Revised: We introduced a generalization of NRE - A and

Original: NRE - B which discourages the NRE - B specific c w ( x ) bias term and produces an unbiased estimate of the likelihood-to-evidence ratio at optimum.
Revised: NRE - B which discourages the NRE - B specific c w ( x ) bias term and produces an unbiased estimate of the likelihood-to-evidence ratio at optimum.

Original: This property has implications for the importance sampling diagnostic, which is critical to practitioners.
Revised: This property has implications for the importance sampling diagnostic, which is critical to practitioners.

Original: We suggested using a variational bound on the mutual information as a model-selection metric that could replace the
Revised: We suggested using a variational bound on the mutual information as a model-selection metric that could replace the

Original: C2ST averaged over several observations.
Revised: C2ST averaged over several observations.

Original: It is significantly more practical since it does not require access to the ground truth posterior; however, it remains a lower bound so we don’t know the overall quality of our estimator.
Revised: It is significantly more practical since it does not require access to the ground truth posterior; however, it remains a lower bound so we don’t know the overall quality of our estimator.

Original: Further discussions and derivations related to the mutual information are available in Appendix D.
Revised: Further discussions and derivations related to the mutual information are available in Appendix D.

Original: In the context of NRE - C , we found that increasing the number of contrastive parameters K improved the C2ST in most cases, at the price of increasing the normalizing constant.
Revised: In the context of NRE - C , we found that increasing the number of contrastive parameters K improved the C2ST in most cases, at the price of increasing the normalizing constant.

Original: Setting γ ≈ 1 was generally the fastest to saturate and lower values seemed to slightly improve performance with the Large NN and fixed simulation data.
Revised: Setting γ ≈ 1 was generally the fastest to saturate and lower values seemed to slightly improve performance with the Large NN and fixed simulation data.

Original: These results indicate that better performance can be achieved by using NRE - C with these hyperparameters than the other version of NRE .
Revised: These results indicate that better performance can be achieved by using NRE - C with these hyperparameters than the other version of NRE .

Original: When both training parameters θ and simulation data x are unlimited, the architecture size plays an important role in the quality of surrogate model.
Revised: When both training parameters θ and simulation data x are unlimited, the architecture size plays an important role in the quality of surrogate model.

Original: We tried drawing contrastive parameters from the prior, a commonly available practical use-case, and found that it damped the effect of K on the C2ST.
Revised: We tried drawing contrastive parameters from the prior, a commonly available practical use-case, and found that it damped the effect of K on the C2ST.

Original: According to the C2ST, for highest accuracy with a fixed budget set K large, up to the mini-batch size divided by 2, and γ ≈ 1 . 0 for the best convergence rate.
Revised: According to the C2ST, for highest accuracy with a fixed budget set K large, up to the mini-batch size divided by 2, and γ ≈ 1 . 0 for the best convergence rate.

Original: When the prior is sampled for every mini-batch, applying a higher K may be less valuable.
Revised: When the prior is sampled for every mini-batch, applying a higher K may be less valuable.

Original: In situations where the normalizing constant should be very close to unity, i.e., for practitioners who want to run diagnostics, drawing θ from the prior with every mini-batch and using a smaller K and γ is best.
Revised: In situations where the normalizing constant should be very close to unity, i.e., for practitioners who want to run diagnostics, drawing θ from the prior with every mini-batch and using a smaller K and γ is best.

Original: Broader Impact
Revised: Broader Impact

Original: The societal implications of SBI are similar to other inference methods.
Revised: The societal implications of SBI are similar to other inference methods.

Original: The methods are primarily scientific and generally lead to interesting discoveries; however, one must be careful to use an accurate generative model and to carefully test empirical results.
Revised: The methods are primarily scientific and generally lead to interesting discoveries; however, one must be careful to use an accurate generative model and to carefully test empirical results.

Original: Model mismatch and untested inference can lead to incorrect conclusions.
Revised: Model mismatch and untested inference can lead to incorrect conclusions.

Original: That is why we emphasize the importance of the diagnostics in our paper.
Revised: That is why we emphasize the importance of the diagnostics in our paper.

Original: This nuance can be missed by practitioners doing inference in any field; however, special care should be taken when producing inference results that may be used for making decisions in areas like predicting hidden variables that describe human behavior or determining what factors are responsible for climate change.
Revised: This nuance can be missed by practitioners doing inference in any field; however, special care should be taken when producing inference results that may be used for making decisions in areas like predicting hidden variables that describe human behavior or determining what factors are responsible for climate change.

Original: This list is non-comprehensive and not specific to SBI .
Revised: This list is non-comprehensive and not specific to SBI .

Original: Acknowledgments and Disclosure of Funding
Revised: Acknowledgments and Disclosure of Funding

Original: We thank Noemi Anau Montel, James Alvey, Kosio Karchev for reading; Teodora Pandeva for mathematics consultation; Maud Canisius for design consultation; Marco Federici and Gilles Louppe for discussions.
Revised: We thank Noemi Anau Montel, James Alvey, Kosio Karchev for reading; Teodora Pandeva for mathematics consultation; Maud Canisius for design consultation; Marco Federici and Gilles Louppe for discussions.

Original: This work uses numpy [26], scipy [72], seaborn [73], matplotlib [32], pandas [52, 74], pytorch [57], and jupyter [38].
Revised: This work uses numpy [26], scipy [72], seaborn [73], matplotlib [32], pandas [52, 74], pytorch [57], and jupyter [38].

Original: We want to thank the DAS-5 computing cluster for access to their TitanX GPUs.
Revised: We want to thank the DAS-5 computing cluster for access to their TitanX GPUs.

Original: DAS-5 is funded by the NWO/NCF (the Netherlands Organization for Scientific Research).
Revised: DAS-5 is funded by the NWO/NCF (the Netherlands Organization for Scientific Research).

Original: We thank SURF (www.surf.nl) for the support in using the Lisa Compute Cluster.
Revised: We thank SURF (www.surf.nl) for the support in using the Lisa Compute Cluster.

Original: Benjamin Kurt Miller is funded by the University of Amsterdam Faculty of Science (FNWI), Informatics Institute (IvI), and the Institute of Physics (IoP).
Revised: Benjamin Kurt Miller is funded by the University of Amsterdam Faculty of Science (FNWI), Informatics Institute (IvI), and the Institute of Physics (IoP).

Original: We received funding from the European Research
Revised: We received funding from the European Research

Original: Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (Grant agreement No. 864035 – UnDark).
Revised: Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (Grant agreement No. 864035 – UnDark).

Original: Checklist
Revised: Checklist

Original: The checklist follows the references.
Revised: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information on how to answer these questions.
Revised: Please read the checklist guidelines carefully for information on how to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
Revised: For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .

Original: You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
Revised: You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.

Original: For example:
Revised: For example:

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [Yes] See Section .
Revised: [Yes] See Section .

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [No] The code and the data are proprietary.
Revised: [No] The code and the data are proprietary.

Original: • Did you include the license to the code and datasets?
Revised: • Did you include the license to the code and datasets?

Original: [N/A]
Revised: [N/A]

Original: Please do not modify the questions and only use the provided macros for your answers.
Revised: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the
Revised: Note that the

Original: Checklist section does not count towards the page limit.
Revised: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
Revised: In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] We introduce the generalization in Section 2.1.
Revised: [Yes] We introduce the generalization in Section 2.1.

Original: We show that it has the properties we claim in Appendix B.
Revised: We show that it has the properties we claim in Appendix B.

Original: We perform the experiments in Section 3 and make a hyperparameter recommendation in Section(b) Did you describe the limitations of your work?
Revised: We perform the experiments in Section 3 and make a hyperparameter recommendation in Section(b) Did you describe the limitations of your work?

Original: [Yes] We discussed the behavior of ours and other algorithms in the finite sample setting, see Section 2.1 and limitations of existing to test the convergence of our method, see Section 2.2.
Revised: [Yes] We discussed the behavior of ours and other algorithms in the finite sample setting, see Section 2.1 and limitations of existing to test the convergence of our method, see Section 2.2.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] We briefly mentioned them in Section 4.
Revised: [Yes] We briefly mentioned them in Section 4.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] This work applies toy simulation data thus no ethics concerns were raised by the paper.
Revised: [Yes] This work applies toy simulation data thus no ethics concerns were raised by the paper.

Original: Other matters of ethics during experimentation and writing conformed.
Revised: Other matters of ethics during experimentation and writing conformed.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes] These are laid out in Appendix B. (b) Did you include complete proofs of all theoretical results?
Revised: [Yes] These are laid out in Appendix B. (b) Did you include complete proofs of all theoretical results?

Original: [Yes] We have only one real theoretical result and that is proven in Lemma 1.
Revised: [Yes] We have only one real theoretical result and that is proven in Lemma 1.

Original: Anything else follows algebraically and is shown in detail.
Revised: Anything else follows algebraically and is shown in detail.

Original: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Original: [Yes] The code will be made available via a link in the supplemental material.
Revised: [Yes] The code will be made available via a link in the supplemental material.

Original: It is a python package with example calls and self-explanatory installation instructions.
Revised: It is a python package with example calls and self-explanatory installation instructions.

Original: The details will be shown in Section C. Results will be released upon acceptance.
Revised: The details will be shown in Section C. Results will be released upon acceptance.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] Additionally to the hyperparameter search which is the primary experimental result, seen in Section 3, we included more details in Appendix C. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
Revised: [Yes] Additionally to the hyperparameter search which is the primary experimental result, seen in Section 3, we included more details in Appendix C. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] The hyperparameter searches indeed included error bars in Section 3.
Revised: [Yes] The hyperparameter searches indeed included error bars in Section 3.

Original: We did not include error bars in the computation of the diagnostic since we emphasize that it is made possible by our method, not its statistical properties.
Revised: We did not include error bars in the computation of the diagnostic since we emphasize that it is made possible by our method, not its statistical properties.

Original: The details of the benchmark, including some uncertainty in the last experimental section are shown in Appendix C. (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: The details of the benchmark, including some uncertainty in the last experimental section are shown in Appendix C. (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] This is discussed in Appendix C. 4.
Revised: [Yes] This is discussed in Appendix C. 4.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] We cited the benchmark [44] several times in the paper.
Revised: [Yes] We cited the benchmark [44] several times in the paper.

Original: Also the sbi package [68]
Revised: Also the sbi package [68]

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [Yes] The license is mentioned in Appendix A (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [Yes] The license is mentioned in Appendix A (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes] I will link to my code in the supplemental material as discussed above.
Revised: [Yes] I will link to my code in the supplemental material as discussed above.

Original: Upon acceptance results will be made available.
Revised: Upon acceptance results will be made available.

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [N/A]
Revised: [N/A]

Original: The publicly available data was used in accordance with the aforementioned license in [44].
Revised: The publicly available data was used in accordance with the aforementioned license in [44].

Original: There was no need to get consent or mention it in the paper.
Revised: There was no need to get consent or mention it in the paper.

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A]
Revised: [N/A]

Original: The data is generated by toy mathematical models which abstractly relate parameters to data.
Revised: The data is generated by toy mathematical models which abstractly relate parameters to data.

Original: It is highly unlikely any culture would find them offensive.
Revised: It is highly unlikely any culture would find them offensive.

Original: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Original: A Relationship to other SBI methods Sampling in SBI

Original: All NRE (and SBI ) algorithms require samples from the joint distribution p ( θ , x ) .

Original: NRE additionally requires samples from the product of marginals p ( θ ) p ( x ) .

Original: Sampling the joint with a simulator requires drawing from the prior θ ∼ p ( θ ) then passing that parameter into the simulator to produce x ∼ p ( x | θ ) .

Original: Sampling the product of marginals is simple, just take another sample from the prior θ ′ ∼ p

Original: ( θ )

Original: and pair it with our simulation from before.

Original: Then we have ( θ , x ) ∼ p ( θ , x ) and ( θ ′ , x ) ∼ p ( θ ) p ( x ) .

Original: In practice, we refer to this operation as a bootstrap within a mini-batch where we take θ ′ from other parameter-simulation pairs and reuse them to create samples drawn from the product of marginals p ( θ ) p ( x ) .

Original: NRE - C sometimes requires more θ samples, often represented as Θ .

Original: These can be generated by merely concatenating several samples from p ( θ ) .

Original: Some SBI methods, e.g., sequential methods, sometimes replace the prior with a proposal distribution ˜ p

Original: ( θ ) .

Original: Amortized and sequential SBI

Original: Recently, significant progress has been made in SBI , especially with so-called sequential methods that use active learning to draw samples from the posterior for a fixed observation-of-interest x o [16, 18, 20, 21, 24, 35, 42, 53, 56, 59, 60, 61, 63, 69].

Original: Amortized

Original: SBI algorithms, that can draw samples from the posterior for arbitrary observation x , have also enjoyed attentive development [8, 9, 30, 36, 47, 48].

Original: Hermans et al.

Original: [31] and Miller et al.

Original: [48] argue that their intrinsic empirical testability makes amortized methods better applicable to the scientific use-case despite their inherently higher training expense.

Original: The last pillar of development has been into assessment methods that determine the reliability of approximate inference results.

Original: In the machine learning community, the focus has been on evaluating the exactness of estimates for tractable problems [44].

Original: Evaluation methods which apply in the practical case where no tractable posterior is available are under development [7, 12, 14, 29, 67].

Original: We make a contribution to this effort in Appendix D.

Original: Contrastive learning, NRE - B , and NPE

Original: We call our method Contrastive Neural Ratio Estimation because it the classifier is trained to identify which pairs of θ and x should be paired together.

Original: Gutmann et al.

Original: [25] created an overview of contrastive learning for statistical problems generally.

Original: Specific connections are in the loss functions with NRE - A closely corresponding to noise-contrastive estimation (NCE) [22, 23] and NRE - B with RankingNCE, InfoNCE, and related [45, 46, 49, 71].

Original: A core aspect of our paper focus on the effects of the NRE - B ratio estimate biased by c w ( x ) at optimum.

Original: Ma and Collins [45] also investigate the effects on the partition function when applying a binary or multi-class loss variant for estimating conditional energy-based models.

Original: Due to the similarity in the loss functions, they exhibit a similar bias in their partition function.

Original: In order to correct this bias they estimate c w ( x ) directly.

Original: We did not attempt to do this for our problem, so we do not have intuition about the effectiveness of such an approach to the likelihood-based diagnostic in NRE .

Original: Although, since c w ( x ) is completely unconstrained, it may be quite difficult to estimate.

Original: We believe that this would be an alternative direction for future work.

Original: Durkan et al.

Original: [16] emphasize the connection between NRE and contrastive learning in their paper which created a framework such that Neural Posterior Estimation ( NPE ) and NRE - B can be trained using the same loss function.

Original: The addition of an independently drawn ( Θ , x ) set means that the NREC framework is not trivially applicable for computing the normalizing constant on atoms necessary for their sequential version of NPE .

Original: Software Our experiments used sbi [68], which is released under an AGPL-3.0 license, and sbibm [44], which is released under an MIT license.

Original: They implement various neural SBI algorithms and benchmark problems respectively.

Original: A.1 Full derivation of other NRE methods using our framework

Original: Here we present a derivation of the previous works NRE - A [30] and NRE - B [30] in our framwork.

Original: NRE - A

Original: To estimate r ( x | θ ) , Hermans et al.

Original: [30] introduce an indicator variable y which switches between dependently and independently drawn samples.

Original: We have conditional probability

Original: [Equation]

Original: Each class’ marginal probability is set equally, p ( y = 0)

Original: =

Original: p ( y = 1) .

Original: Dropping the NRE - A subscript, the probability that ( θ , x ) was drawn jointly is encoded in the another conditional probability

Original: [Equation]

Original: [Equation]

Original: NRE - A estimates log ˆ r ( x | θ ) with neural network f w with weights w .

Original: Training is done by minimizing the binary cross-entropy of q w ( y = 1 | θ , x ) :

Original: = σ ◦ f w ( θ , x ) relative to p ( y, θ , x ) .

Original: For B samples, w = arg min w

Original: [Equation]

Original: where θ ( b ) , x ( b ) ∼ p ( θ ) p ( x ) and θ ( b ′ ) , x ( b ′ ) ∼ p ( θ , x ) .

Original: In practice, θ ( b ′ ) is bootstrapped from within the mini-batch to produce θ ( b ) .

Original: NRE - A ’s ratio estimate converges to f w = log p ( x | θ ) p ( x )

Original: = log r ( x | θ ) given unlimited model flexibility and data

Original: NRE - B

Original: Durkan et al.

Original: [16] estimate r ( x | θ ) by training a classifier that selects from among K parameters Θ := ( θ 1 , . .

Original: . , θ K ) which could have generated x .

Original: In contrast with NRE - A , one of these parameters θ k is always drawn jointly with x .

Original: Let y be a random variable which indicates which one of K parameters simulated x .

Original: The marginal probability p ( y = k ) :

Original: = 1 /K is uniform.

Original: That means

Original: [Equation]

Original: defines our conditional probability for parameters and data.

Original: Bayes’ rule reveals a conditional distribution over y , dropping the NRE - B subscript, therefore p

Original: [Equation]

Original: [Equation]

Original: NRE - B estimates log ˆ r ( x | θ ) with a neural network g w .

Original: Training is done by minimizing the cross entropy of q w ( y = k | Θ , x ) :=

Original: exp ◦ g w ( θ k , x ) / (cid:80) i exp ◦ g w ( θ i , x ) relative to p ( y, x , θ ) ;

Original: [Equation]

Original: where θ ( b ′ ) 1 , . .

Original: , θ ( b ′ )

Original: K ∼ p

Original: ( θ ) , k ( b ′ )

Original: ∼ p

Original: ( y ) , and x ( b ′ )

Original: ∼ p ( x | θ ( b ′ ) k ) over B samples.

Original: In our parameterization and given unlimited flexibility and data, g w ( θ , x ) = log p ( θ | x ) p ( θ ) + c w ( x ) at convergence.

Original: The extra term enters because the optimal classifier for (22) need not be normalized.

Original: We present first the proof of convergence for NRE - C . Afterwards, we discuss the properties of estimated importance weights in NRE - B . B.1 Proof of convergence of NRE - C
Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: Lemma 1.

Original: Consider for k = 0 , . .

Original: , K the following probability distributions for z :

Original: [Equation]

Original: [-23) and p ( y ) > 
-]
Revised: [-23) and p ( y ) > 
-]

Original: a probability distribution for y .

Original: Put p k :=

Original: p ( y = k ) for k = 1 , . .

Original: , K .

Original: For functions f k : Z → R , k = 1 , . . .

Original: , K , let:

Original: [Equation]

Original: Note that q ( y = k | f, z ) > 0 for all k = 0 , . .

Original: , K and that (cid:80)

Original: [-Kk =
-]
Revised: [-Kk =
-]

Original: q ( y = k | f, z )

Original: = 1 for every

Original: K -tuple f

Original: = ( f k ) k

Original: =1 ,...,K and z ∈ Z .

Original: Consider a minimizer:

Original: [Equation]

Original: [Equation]

Original: Then we have for p ( z ) -almost-all z and all k = 1 , . .

Original: , K :

Original: [Equation]

Original: Proof.

Original: We have:

Original: [Equation]

Original: [Equation]

Original: which is minimized, when KLD = 0 , thus:

Original: [Equation]

Original: which implies that for p ( z ) -almost-all z :

Original: [Equation]

Original: So we get with the definition of q ( y | f ∗ , z ) :

Original: [Equation]

Original: Dividing the latter by the former gives for k = 1 , . .

Original: , K :

Original: [Equation]

Original: implying for k = 1 , . .

Original: , K and p ( z ) -almost-all z :

Original: [Equation]

Original: This shows the claim.

Original: We used the symbol KLD to imply the Kullback-Leibler Divergence.

Original: The proof in Lemma 1 is slightly more general than necessary for our typical case.

Original: All our functions f k are typically the same, namely the evaluation of a neural network with weights w .

Original: Rather than searching for the function f which minimizes the objective, we search for the weights, but these are equivalent.

Original: Finally to make everything fit, we set z :=

Original: ( θ , x ) .

Original: B.2 Properties of the importance sampling diagnostic on biased ratio estimates

Original: In Section 2.2 we discuss the importance sampling diagnostic, in which the estimated ratio is tested by comparing weighted samples from x ∼ p ( x ) to unweighted samples from x ∼ p ( x | θ ) .

Original: The estimated ratio from NRE - C is merely exp( h w ( θ , x )) and is therefore not restricted to have the properties of a ratio of probability distributions, except at optimum with unlimited flexibility and data.

Original: The importance sampling diagnostic is designed to test whether the estimated ratio is close to having these properties for a fixed θ .

Original: One way to improve ratios estimated by NRE - C is to compute the normalizing constant Z w ( x ) , which should be close to one, and replace the ratio with this “normalized” version.

Original: The unrestricted nature of the NRE - B -specific c w ( x ) bias means that the normalization constant

Original: Z w ( x ) does not have to be close to one.

Original: Further, we show that the unrestricted bias means that the normalizing constant for two NRE - B ratio estimators, which are equivalent in terms of their loss function (22), is not unique, i.e., the estimate is ill-posed.

Original: This property means that normalizing

Original: NRE - B will not, in general, produce an estimator which passes the diagnostic.

Original: Consider two ratio estimates with the relationship ˆ r 1 ( x | θ )

Original: =

Original: ˆ r 2 ( x | θ ) /C

Original: ( x ) where C is an positive function of x resulting from the aggregation of the exponentiated bias.

Original: Given N

Original: samples

Original: x n ∼ p ( x ) with weights w 1 ( x n ) =

Original: ˆ r 1 ( x n | θ ) and w 2 ( x n ) =

Original: ˆ r 2 ( x n | θ ) , we can compute the importancenormalized weights by

Original: [Equation]

Original: However, if we substitute this constant back into our expression, we find that the weights do not agree

Original: [Equation]

Original: Therefore, normalization does not “protect” against scaling bias introduced by a function in x .

Original: NRE - B does not penalize functional biases like C ( x ) , so even ratios considered optimal by NRE - B can easily fail the diagnostic.

Original: Meanwhile, NRE - C encourages terms like C ( x ) towards one.

Original: That has the effect of making the C ( x ) ≈ 1 drop out of the normalized importance weights, thus making performance on the diagnostic indicative of a better ratio estimate, given enough classifier flexibility.

Original: C Experimental Details

Original: Computational costs Our experiments were performed on a cluster of Nvidia Titan V graphics processing units.

Original: The primary expense was the hyperparameter search within Sections 3.1, 3.2, and the first part of Section 3.3.

Original: The run of those experiments took about 50,000 gpu-minutes total considering all of our current data.

Original: Since there were several iterations, we multiply this by three to estimate total compute.

Original: The next expense was the computation of sbibm in Section 3.3 that took about 24,000 gpu-minutes.

Original: Together these equal about 2880 gpu-hours.

Original: An estimate for the total carbon contributions corresponds to 311.04 kgCO 2 .

Original: Luckily, our clusters run completely on wind power offsetting the contribution.

Original: Estimations were conducted using the MachineLearning Impact calculator presented in [39].

Original: Architecture and training The centerpiece of our method is the neural network h w which is trained as a classifier.

Original: The hyperparameter choices here were fairly constant throughout the experiments.

Original: Any hyperparameters for training or architecture which were consistent across experiments are listed in Table 2.

Original: Hidden features and number of RESNET blocks depend on the architecture.

Original: Large NN uses three resnet blocks with 128 features, while Small NN uses two resnet blocks and 50 features.

Original: [Table]

Original: C.1 Hyperparameter search measured with C2ST

Original: In this section, we trained many neural networks with different hyperparameter settings on three different tasks from Lueckmann et al.

Original: We trained both architectures, Large and Small NN, on a grid of γ and K values with NRE - B and NRE - C .

Original: No matter whether we were training with unlimited draws from the joint, prior, or fixed data we designed an epoch such that it has 20 training mini-batches and 2 validation mini-batches.

Original: For fixed initial data and prior this corresponds to a simulation budget of 22,528.

Original: The mean validation losses per-epoch for these networks is visible in

Original: Figures 7, 8, and 9.

Original: For one specific problem, we also computed validation loss at a fixed γ and K no matter the training regime in an effort to produce comparable validation losses, specifically γ = 1 and K = 1 aka

Original: the

Original: NRE - A regime.

Original: This training setting reverses some of the trends we’ve seen when validated on the same loss as training data.

Original: We did not apply it further since this could bias results.

Original: See Figure 11.

Original: To compare, we also plot the validation loss on the sample problem but using the γ and K each model was trained with.

Original: See Figure 10, note that the bias depending on γ and K is clearly visible (just as in the other validation loss plots).

Original: Once the networks were trained, we drew samples from ten per-task posteriors based on ten predefined observations

Original: x i with i ∈ 1 , 2 , . . .

Original: This leveraged the amortized property of NRE - C .

Original: Samples were drawn on these problems using rejection sampling.

Original: Once the samples were drawn, they were compared to ground truth posterior samples from the benchmark with the C2ST.

Original: The detailed plot which shows per-task behavior is available in Figure 6.

Original: [Figure]

Original: (a) Draw ( θ , x ) .

Original: [Figure]

Original: (b) Draw θ

Original: [Figure]

Original: Recall that assigns to inaccurate

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: where colors indicate different contrastive parameter counts

Original: , i

Original: [Figure]

Original: e, K + 1 .

Original: A fixed validation loss was used, namely ℓ 1 , 1 .

Original: Although the validation loss is now comparable so that we can see the performance of the different classifiers on the same task, some of the convergence rate trends disagreed with SLCP in Figure 10.

Original: Naturally, the classifier trained to distinguish two samples often performed best on this task.

Original: C.2 Simulation-based inference benchmark

Original: [-We trained NRE - C with γ = 1 . 
-]
Revised: [-We trained NRE - C with γ = 1 . 
-]

Original: and

Original: K = 100 using Large NN for 1000 epochs on all of the tasks from the simulation-based inference benchmark [44].

Original: In contrast with our previous hyperparameter search experiments, we used their Markov-chain Monte Carlo sampling scheme which applies slice sampling [51].

Original: The details results are presented in Table 3 and Table 4.

Original: There were three tasks which did not succeed with this sampling procedure, there we used rejection sampling.

Original: Our training slightly diverged from the benchmark.

Original: We enumerate the ways below.

Original: • Large NN is a bigger architecture than the one which produced their reported NRE - B values.

Original: In fact, the NRE - B version has the same settings as Small NN; however, we found that larger networks performed better in our search and chose the bigger one for that reason.

Original: • In the benchmark, networks are trained with early stopping but we always trained forepochs.

Original: We selected the network which performed best on the validation loss, using the same values for K and γ in the validation loss as in the training.

Original: • Even though NRE - B is an amortized method, the benchmark trained a network for every observation.

Original: Instead, we leveraged the amortization properties of NRE - C and trained a single network which drew samples from each approximate posterior given by each of the ten observations.

Original: Task details We provide a short summary of all of the inference tasks in the SBI benchmark by

Original: Lueckmann et al.

Original: Bernoulli GLM

Original: This task is a generalized linear model.

Original: The likelihood is Bernoulli distributed.

Original: The data is a 10-dimensional summary statistic from an 100-dimensional raw vector.

Original: The posterior is 10-dimensional and it only has one mode.

Original: Bernoulli GLM Raw This is the same task as above, but instead the entire 100-dimensional observation is shown to the inference method rather than the summary statistic.

Original: Gaussian Linear

Original: A simple task with a Gaussian distributed

Original: prior

Original: and a

Original: Gaussian likelihood

Original: over the mean.

Original: Both have a Σ = 0 .

Original: I covariance matrix.

Original: The posterior is also Gaussian.

Original: It is performed in 10-dimensions for the observations and parameters.

Original: Gaussian Linear Uniform This is the same

Original: as the task above, but instead the prior over the mean is a 10-dimensional uniform distribution from -1 to 1 in every dimension.

Original: Gaussian Mixture

Original: This task occurs in the ABC literature often.

Original: [-Infer the common mean of a mixture of Gaussians where one has covariance matrix Σ = 1 . 
-]
Revised: [-Infer the common mean of a mixture of Gaussians where one has covariance matrix Σ = 1 . 
-]

Original: · I and the other Σ = 0 .

Original: · I .

Original: It occurs in two dimensions.

Original: Lotka Volterra This is an ecological predator-prey model where the simulations are generated from randomly drawn initial conditions by solving a parameterized differential equation.

Original: There are four parameters that control the coupling between the generation and destruction of both prey and predators.

Original: The priors are log normal.

Original: The data is a twenty dimensional summary statistic.

Original: SIR An epidemiological model simulating the progress of an contagious disease outbreak through a population.

Original: Simulations are generated from randomly drawn initial conditions with a parameterized differential equation defining the dynamics.

Original: There are two parameters with a log normal prior.

Original: The data is a ten dimensional summary statistic.

Original: SLCP A task which has a very simple non-spherical Gaussian likelihood, but a complex posterior over the five parameters which, via a non-linear function, define the mean and covariance of the likelihood.

Original: There are five parameters each with a uniform prior from -3 to 3.

Original: The data is four-dimensional but we take two samples from it.

Original: It was introduced in [56].

Original: SLCP with Distractors This is the same task as above but instead the data is concatenated with 92 dimensions of Gaussian noise.

Original: Two Moons

Original: This task exhibits a crescent shape posterior with bi-modality–two of the attributes often used to stump MCMC samplers.

Original: Both the data and parameters are two dimensional.

Original: The prior is uniform from -1 to 1.

Original: [Table]

Original: SMC - ABC 0.

Original: SNLE 0.

Original: SNPE 0.

Original: SNRE ( SNRE - B ) 0.

Original: Gaussian Linear Uniform

Original: NRE - C (ours) 0.

Original: REJ - ABC 0.

Original: NLE 0.

Original: NPE 0.

Original: NRE ( NRE - B ) 0.

Original: SMC - ABC 0.

Original: SNLE 0.

Original: SNPE 0.

Original: SNRE ( SNRE - B ) 0.

Original: Gaussian Mixture

Original: NRE - C (ours) 0.

Original: REJ - ABC 0.

Original: NLE 0.

Original: NPE 0.

Original: NRE ( NRE - B ) 0.

Original: SMC - ABC 0.

Original: SNLE 0.

Original: SNPE 0.

Original: SNRE ( SNRE - B ) 0.

Original: [Table]

Original: - ABC 0.

Original: SNLE 0.

Original: SNPE 0.

Original: SNRE ( SNRE - B ) 0.

Original: SLCP Distractors

Original: NRE - C (ours) 0.

Original: REJ - ABC 0.

Original: NLE 0.

Original: NPE 0.

Original: NRE ( NRE - B ) 0.

Original: SMC - ABC 0.

Original: SNLE 0.

Original: SNPE 0.

Original: SNRE ( SNRE - B ) 0.

Original: Two Moons

Original: NRE - C (ours) 0.

Original: REJ - ABC 0.

Original: NLE 0.

Original: NPE 0.

Original: NRE ( NRE - B ) 0.

Original: SMC - ABC 0.

Original: SNLE 0.

Original: SNPE 0.

Original: SNRE ( SNRE - B ) 0.

Original: D Mutual Information

Original: Estimating the mutual information is closely related to estimating the likelihood-to-evidence ratio.

Original: We reference various bounds on the mutual information and show how NRE - C can estimate them numerically.

Original: These bounds obey the variational principle and might be a practical candidate for validating the performance of SBI methods for scientific purposes, i.e., when the ground truth posterior is intractable.

Original: The approximation of the mutual information could synergize with other diagnostics like empirical, expected coverage testing and the importance sampling diagnostic.

Original: See section 2.2.

Original: Mutual information and expected Kullback-Leibler divergence If p ( θ | x ) denotes the true posterior and p w ( θ | x ) an approximate posterior then the quality of that approximation can be measured via the (forward) Kullback-Leibler divergence:

Original: [Equation]

Original: However, in the SBI -setting we have only access to the likelihood p ( x | θ ) via samples.

Original: Since we have also access to the prior p ( θ ) (analytically and) via samples we can sample from the joint p ( θ , x ) .

Original: So using the expected Kullback-Leibler divergence is more tractable in the SBI setting to measure the discrepancy between true and approximated prior:

Original: [Equation] In all considered cases in this paper the approximate posterior p w ( θ | x ) is given by: [Equation] where ˆ r w ( x | θ ) comes from a (trained) neural network andZ w ( x ) denotes the normalization constant.
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: With the above notations we get for the expected Kullback-Leibler divergence the expression:

Original: [Equation]

Original: where I ( θ ; x ) is the mutual information w

Original: [Equation]

Original: r

Original: [Equation]

Original: t. p ( θ , x ) .

Original: Since we aim at minimizing the expected

Original: Kullback-Leibler divergence we implicitely aim to maximize our mutual information approximation:

Original: [Equation]

Original: [Equation]

Original: which can be estimated via Monte-Carlo by sampling i

Original: [Equation]

Original: i

Original: [Equation]

Original: d. θ n , θ n,m ∼ p

Original: ( θ ) , x n ∼ p ( x | θ n ) , n = 1 , . . .

Original: , N , m = 1 , . . .

Original: , M and then compute:

Original: [Equation]

Original: Since in all mentioned methods ˆ r w ( x | θ ) is meant to approximate the ratio p ( θ | x ) p ( θ ) and estimating the normalizing constant is expensive a naive alternative to approximate the expected Kullback-Leibler divergence is by naively plugging the unnormalized distribution ˆ q w ( θ | x ) :

Original: = ˆ r w ( x | θ ) p

Original: ( θ ) into the above formula and using the estimate:

Original: [Equation]

Original: While this is justified for the training objectives for NRE - A and NRE - C , which encourage a trivial normalizing constant Z w ( x ) ≈ 1 at optimum, the same is not true for NRE - B , which leads to an additional non-vanishing, possibly arbitrarily big, bias term:

Original: [Equation]

Original: Another way to address the normalizing constant is the use of the Kullback-Leibler divergence that also works for unnormalized distributions p ( z ) , q ( z ) :

Original: [Equation]

Original: which is always ≥ 0 with equality if p ( z )

Original: =

Original: q ( z ) for p ( z ) -almost-all z .

Original: This gives for the expected Kullback-Leibler divergence between the posterior p ( θ | x ) and the unnormalized approximate posterior

Original: ˆ q w ( θ | x

Original: [Equation]

Original: [Equation]

Original: ( x

Original: [Equation]

Original: p (

Original: [Equation]

Original: [Equation]

Original: We see that the normalizing constant Z w ( x ) re-appears, but with a different term.

Original: Similar to before the above can be used for another mutual information approximation given by:

Original: [Equation]
Revised: [N/A]

Original: which can be estimated via Monte-Carlo, again, by sampling i.i

Original: [Equation]

Original: d. θ n , θ n,m ∼ p

Original: ( θ ) , x n ∼ p ( x | θ n ) , n = 1 , . . .

Original: , N , m = 1 , . . .

Original: , M and then computing:

Original: [Equation]

Original: Note that since log( r ) ≤ r − 1 we always have the inequalities:

Original: [Equation]

Original: showing that I (0) w ( θ ; x ) leads to a tighter approximation to the mutual information I ( θ ; x ) than

Original: I (1) w ( θ ; x ) .

Original: The procedures above require estimating the normalizing constant or the partition function.

Original: Generally this can be quite expensive and may require techniques like Nested Sampling [65]; however, it is tractable with Monte Carlo on problems with parameters within low dimensional compact regions.

Original: Bounds on the mutual information There is a connection between the training objective of NRE - B and a multi-sample lower bound on the mutual information [58], as noted in Durkan et al.

Original: Contrastive learning has been explored for estimating the mutual information by Van den Oord et al.

Original: [71] also discussed by Belghazi et al.

Original: The bounds we define above are also discussed in detail by Poole et al.

Original: [58], although we find computing

Original: ˆ I (0) w ( θ ; x ) and ˆ I (1) w ( θ ; x ) to be tractable within

Original: SBI –although potentially expensive.

Original: We attempted to train a ratio estimator by minimizing ˆ I (1) w ( θ ; x ) directly on fixed data.

Original: Our preliminary experiments found that the C2ST was near unity on the SLCP task and other works claim that this mutual information bound has extremely high variance.

Original: We therefore ended our investigation.

Original: Numerical estimates of bounds on mutual information In remains unclear how to evaluate the performance of SBI algorithms across model types without access to the ground truth.

Original: Computing −

Original: ˆ I (0) w ( θ ; x ) as a validation loss is applicable to NRE - B and NRE - C for all γ and K .

Original: Therefore, we investigate estimating this bound on the mutual information for model comparison and as a surrogate for computing the C2ST across several pieces of simulated data.

Original: (It is also noteworthy that this bound is applicable to Neural Posterior Estimation, where the likelihood-to-evidence ratio would be approximated by p w ( θ | x ) /p

Original: ( θ ) and p w ( θ | x ) represents an approximate posterior density.

Original: However, this case is not investigated further in this work.)

Original: In the effort to find a model comparison metric that applies when the user does not have access to the ground truth, we ran another set of experiments where we trained ratio estimators using NRE - B and NRE - C over various γ and K , then we validated them on held-out data with − ˆ I (0) w ( θ ; x ) as a validation loss.

Original: All networks corresponded with the Large NN architecture.

Original: Just like in the main experiments, we computed the C2ST over the ten different observations from the SBI benchmark.

Original: The results of the training can be seen in Figure 4 and in full in Figure 12.

Original: Visually, − ˆ I (0) w ( θ ; x ) was more comparable across models than plotting their classification validation loss, see Figure 10, for both NRE - B and NRE - C .

Original: The classification validation loss has a bias depending on γ and K which − ˆ I (0) w ( θ ; x ) does not exhibit.

Original: The metric therefore allows us to compare models such that model producing the most negative mutual information bound also most-accurately estimates the posterior, on average.

Original: A correlation plot showing the relationship between ˆ I (0) w ( θ ; x ) and the C2ST for various γ and K on the SLCP task can be found in Figure 13.

Original: We find that the two measurements are well correlated, implying that −

Original: ˆ I (0) w ( θ ; x ) , which does not require access to the ground truth posterior, may be able to replace computing the C2ST across several pieces of data, which does require access to the ground truth posterior, but further investigation is necessary.

Original: [-10 0 10 1 10 2 10 3 Epochs2 
-]
Revised: [-10 0 10 1 10 2 10 3 Epochs2 
-]

Original: I ( 0 ) w ( ; x )

Original: NRE

Original: [Figure]

Original: B

Original: [Figure]

Original: [-10101010 3 Epochs2 
-]
Revised: [-10101010 3 Epochs2 
-]

Original: I ( 0 ) w ( ; x ) (gamma) = 0.

Original: 10101010 3 Epochs (gamma) = 0.

Original: 10101010 3 Epochs (gamma) = 0.

Original: 10101010 3 Epochs (gamma) = 1.

Original: 10101010 3 Epochs (gamma) = 10.

Original: 10101010 3 Epochs (

Original: [Figure]

Original: ) =

Original: [Figure]

Original: [Figure]

Original: Numerical estimates of the partition function In addition to bounds on the mutual information,

Original: we computed a Monte Carlo estimate of the partition function, Z w ( x ) , based on data from the validation set at every epoch during training.

Original: The value of the estimated partition function as a function of epoch is shown in Figure 14 for this set of runs on the SLCP task.

Original: The estimated Z w ( x ) , based on the ratio from NRE - B , is completely unconstrained and varies significantly with epoch.

Original: NRE - C does encourage the partition function to remain “near” unity, although both γ and K affect the strength of the encouragement.

Original: We find that large numbers of contrastive parameters cause Z w ( x ) to deviate significantly from unity; although, often by tens of orders of magnitude less than NRE - B .

Original: This result is connected to the reliability of the importance sampling diagnostic, see Section 2.2.

Original: If the partition function is not near unity, then the estimated likelihood-to-evidence ratio does not cancel with the evidence and the diagnostic will behave like NRE - B , i.e., it becomes possible to produce accurate, albeit unnormalized, posteriors while failing the diagnostic.

Original: This is one reason to limit the number of contrastive parameters.

Original: [Figure]

Original: Just like in the fixed data regime in the main experiment, see Figure 6, we found that on this task a lower γ improved the C2ST.

Original: We also find that the mutual information is correlated with the average

Original: C2ST across 10 pieces of data, but the mutual information has the practical advantage that we can bound it without knowing the ground truth posterior.

Original: The C2ST requires being able to sample from the ground truth posterior.

Original: This data represents the same set of experiments as in Figure 4, Figure 12, and Figure 14.

Original: [Figure]

Original: Z w ( x ) is completely unconstrained with NRE - B ; however, it is constrained with NRE - C .

Original: The strength of encouragement the partition function towards unity depends on the hyperparameters for NRE - C .

Original: ˆ

Original: Z w ( x ) can still take on extreme values, especially for large number of contrastive examples and large γ .

Original: This represents the same set of experiments as in Figure 4, Figure 4 and Figure 13.