Original: Robust Neural Posterior Estimation and Statistical Model Criticism
Revised: Robust Neural Posterior Estimation and Statistical Model Criticism

Original: Abstract
Revised: Abstract

Original: Computer simulations have proven a valuable tool for understanding complex phenomena across the sciences.
Revised: Computer simulations have proven a valuable tool for understanding complex phenomena across the sciences.

Original: However, the utility of simulators for modelling and forecasting purposes is often restricted by low data quality, as well as practical limits to model fidelity.
Revised: However, the utility of simulators for modelling and forecasting purposes is often restricted by low data quality, as well as practical limits to model fidelity.

Original: In order to circumvent these difficulties, we argue that modellers must treat simulators as idealistic representations of the true data generating process, and consequently should thoughtfully consider the risk of model misspecification .
Revised: In order to circumvent these difficulties, we argue that modellers must treat simulators as idealistic representations of the true data generating process, and consequently should thoughtfully consider the risk of model misspecification .

Original: In this work we revisit neural posterior estimation ( NPE ), a class of algorithms that enable black-box parameter inference in simulation models, and consider the implication of a simulation-to-reality gap.
Revised: In this work we revisit neural posterior estimation ( NPE ), a class of algorithms that enable black-box parameter inference in simulation models, and consider the implication of a simulation-to-reality gap.

Original: While recent works have demonstrated reliable performance of these methods, the analyses have been performed using synthetic data generated by the simulator model itself, and have therefore only addressed the well-specified case.
Revised: While recent works have demonstrated reliable performance of these methods, the analyses have been performed using synthetic data generated by the simulator model itself, and have therefore only addressed the well-specified case.

Original: In this paper, we find that the presence of misspecification, in contrast, leads to unreliable inference when NPE is used naïvely.
Revised: In this paper, we find that the presence of misspecification, in contrast, leads to unreliable inference when NPE is used naïvely.

Original: As a remedy we argue that principled scientific inquiry with simulators should incorporate a model criticism component, to facilitate interpretable identification of misspecification and a robust inference component, to fit ‘wrong but useful’ models.
Revised: As a remedy we argue that principled scientific inquiry with simulators should incorporate a model criticism component, to facilitate interpretable identification of misspecification and a robust inference component, to fit ‘wrong but useful’ models.

Original: We propose robust neural posterior estimation ( RNPE ), an extension of NPE to simultaneously achieve both these aims, through explicitly modelling the discrepancies between simulations and the observed data.
Revised: We propose robust neural posterior estimation ( RNPE ), an extension of NPE to simultaneously achieve both these aims, through explicitly modelling the discrepancies between simulations and the observed data.

Original: We assess the approach on a range of artificially misspecified examples, and find RNPE performs well across the tasks, whereas naïvely using NPE leads to misleading and erratic posteriors.
Revised: We assess the approach on a range of artificially misspecified examples, and find RNPE performs well across the tasks, whereas naïvely using NPE leads to misleading and erratic posteriors.

Original:  Introduction
Revised:  Introduction

Original: Stochastic simulators have become a ubiquitous modelling tool across the sciences and are regularly applied to some of the most complex and challenging problems of scientific interest, including climate change (see e.g. Randall et al., 2007), particle physics (e.g. Brehmer et al., 2020) and the Covidpandemic (e.g. Ferguson et al., 2020).
Revised: Stochastic simulators have become a ubiquitous modelling tool across the sciences and are regularly applied to some of the most complex and challenging problems of scientific interest, including climate change (see e.g. Randall et al., 2007), particle physics (e.g. Brehmer et al., 2020) and the Covidpandemic (e.g. Ferguson et al., 2020).

Original: Simulators implicitly define a likelihood function p ( x | θ ) , where x is the simulator output and θ are the simulator parameters.
Revised: Simulators implicitly define a likelihood function p ( x | θ ) , where x is the simulator output and θ are the simulator parameters.

Original: Although running the simulator to sample from the model is straightforward, the inherent complexity of simulators often makes analytic calculation of the likelihood intractable.
Revised: Although running the simulator to sample from the model is straightforward, the inherent complexity of simulators often makes analytic calculation of the likelihood intractable.

Original: As a result, classical inference techniques to find the parameter posterior p ( θ | x ) such as Markov chain Monte Carlo ( MCMC ) (Metropolis et al., 1953) are infeasible.
Revised: As a result, classical inference techniques to find the parameter posterior p ( θ | x ) such as Markov chain Monte Carlo ( MCMC ) (Metropolis et al., 1953) are infeasible.

Original: To overcome this issue, a large family of simulation-based inference ( SBI ) methods have been developed that allow parameter inference to be performed on arbitrary black-box simulators (see Cranmer et al., 2020).
Revised: To overcome this issue, a large family of simulation-based inference ( SBI ) methods have been developed that allow parameter inference to be performed on arbitrary black-box simulators (see Cranmer et al., 2020).

Original: Broadly, these approaches estimate a function that allows access to an approximate posterior.
Revised: Broadly, these approaches estimate a function that allows access to an approximate posterior.

Original: This can be achieved by approximating the posterior directly (Papamakarios and Murray, 2016; Greenberg et al., 2019; Lueckmann et al., 2017), or indirectly, via approximating the likelihood (Papamakarios et al., 2019b) or likelihood-to-evidence ratio
Revised: This can be achieved by approximating the posterior directly (Papamakarios and Murray, 2016; Greenberg et al., 2019; Lueckmann et al., 2017), or indirectly, via approximating the likelihood (Papamakarios et al., 2019b) or likelihood-to-evidence ratio

Original: (Hermans et al., 2020;
Revised: (Hermans et al., 2020;

Original: Thomas et al., 2022), from which the posterior can be sampled using MCMC .
Revised: Thomas et al., 2022), from which the posterior can be sampled using MCMC .

Original: For estimating the posterior or likelihood, neural posterior estimation ( NPE ) and neural likelihood estimation ( NLE ) have shown to be powerful approaches (Lueckmann et al., 2021), which rely on neural network-based conditional density estimators, such as normalising flows, to approximate the likelihood or posterior (Papamakarios et al., 2019a).
Revised: For estimating the posterior or likelihood, neural posterior estimation ( NPE ) and neural likelihood estimation ( NLE ) have shown to be powerful approaches (Lueckmann et al., 2021), which rely on neural network-based conditional density estimators, such as normalising flows, to approximate the likelihood or posterior (Papamakarios et al., 2019a).

Original: [Figure]
Revised: [Figure]

Original: To define misspecification, it is necessary to make a clear distinction between the true data generating process and the simulator: we will write y o for observed data, following some unknown data generating process with distribution denoted p ∗ , and we will write x ∼ p ( x | θ ) to denote samples from the simulator.
Revised: To define misspecification, it is necessary to make a clear distinction between the true data generating process and the simulator: we will write y o for observed data, following some unknown data generating process with distribution denoted p ∗ , and we will write x ∼ p ( x | θ ) to denote samples from the simulator.

Original: In many applications of SBI , summary statistics are used to reduce the dimensionality of the problem.
Revised: In many applications of SBI , summary statistics are used to reduce the dimensionality of the problem.

Original: If this is the case, we use x and y o to denote the simulated and observed summary statistics, rather than the raw data, and treat the raw data as an unobserved latent variable of the model.
Revised: If this is the case, we use x and y o to denote the simulated and observed summary statistics, rather than the raw data, and treat the raw data as an unobserved latent variable of the model.

Original: The simulator is said to be misspecified if the true data generating process does not fall within the family of distributions defined by the simulator, i.e. p ∗ / ∈ { p ( x | θ ); θ ∈ Θ } .
Revised: The simulator is said to be misspecified if the true data generating process does not fall within the family of distributions defined by the simulator, i.e. p ∗ / ∈ { p ( x | θ ); θ ∈ Θ } .

Original: Simulators generally form idealised and simplistic representations of a more complex data generating process, and hence – like all statistical models – will be wrong to some extent.
Revised: Simulators generally form idealised and simplistic representations of a more complex data generating process, and hence – like all statistical models – will be wrong to some extent.

Original: This misspecification often results in a discrepancy between simulated and observed data, sometimes termed a “simulation-to-reality gap” (e.g. Miglino et al., 1995).
Revised: This misspecification often results in a discrepancy between simulated and observed data, sometimes termed a “simulation-to-reality gap” (e.g. Miglino et al., 1995).

Original: Despite this, the general approach in SBI is to learn a posterior approximation q ( θ | x ) (either directly, or indirectly), using simulated data, ( x , θ ) ∼ p ( x | θ ) p ( θ ) , and to condition on the observed data, q ( θ | x = y o ) .
Revised: Despite this, the general approach in SBI is to learn a posterior approximation q ( θ | x ) (either directly, or indirectly), using simulated data, ( x , θ ) ∼ p ( x | θ ) p ( θ ) , and to condition on the observed data, q ( θ | x = y o ) .

Original: This approach is only theoretically justified under the assumption that the simulator is well-specified; relying on this approach for misspecified simulators leads to two issues:
Revised: This approach is only theoretically justified under the assumption that the simulator is well-specified; relying on this approach for misspecified simulators leads to two issues:

Original: (i)
Revised: (i)

Original: As the posterior approximation q ( θ | x ) is learned using samples from the simulator, we can only expect it to form a reasonable approximation of p ( θ | x ) for regions well covered by simulations.
Revised: As the posterior approximation q ( θ | x ) is learned using samples from the simulator, we can only expect it to form a reasonable approximation of p ( θ | x ) for regions well covered by simulations.

Original: For misspecified simulators, the observed data y o may be very unlikely in p ( x ) =
Revised: For misspecified simulators, the observed data y o may be very unlikely in p ( x ) =

Original: (cid:82) p ( x | θ ) p
Revised: (cid:82) p ( x | θ ) p

Original: ( θ ) d θ , in which case the approximation q ( θ | x = y o ) often becomes poor (Cannon et al., 2022).
Revised: ( θ ) d θ , in which case the approximation q ( θ | x = y o ) often becomes poor (Cannon et al., 2022).

Original: In some cases, y o may even fall outside the support of p ( x ) , which results in NPE attempting to extrapolate to estimate the undefined posterior.
Revised: In some cases, y o may even fall outside the support of p ( x ) , which results in NPE attempting to extrapolate to estimate the undefined posterior.

Original: (ii) It has been demonstrated more generally that Bayesian inference is frequently problematic under misspecification.
Revised: (ii) It has been demonstrated more generally that Bayesian inference is frequently problematic under misspecification.

Original: Even for simple models, misspecification can lead to posterior concentration around “bad” models (Grünwald and Langford, 2007; Grünwald and Van Ommen, 2017) and poorly calibrated credible regions (Kleijn and van der Vaart, 2012; Syring and
Revised: Even for simple models, misspecification can lead to posterior concentration around “bad” models (Grünwald and Langford, 2007; Grünwald and Van Ommen, 2017) and poorly calibrated credible regions (Kleijn and van der Vaart, 2012; Syring and

Original: Martin, 2019).
Revised: Martin, 2019).

Original: These issues will persist even in the unlikely case that q ( θ | x ) perfectly approximates p ( θ | x ) .
Revised: These issues will persist even in the unlikely case that q ( θ | x ) perfectly approximates p ( θ | x ) .

Original: One approach to improve robustness is to incorporate an error model to explicitly model potential discrepancies between the observed data and simulations.
Revised: One approach to improve robustness is to incorporate an error model to explicitly model potential discrepancies between the observed data and simulations.

Original: A simple method would be to directly incorporate the error model within the simulator itself, such as by applying additive noise to the output to ensure that the observed data has reasonable marginal likelihood under the model (Cranmer et al., 2020).
Revised: A simple method would be to directly incorporate the error model within the simulator itself, such as by applying additive noise to the output to ensure that the observed data has reasonable marginal likelihood under the model (Cranmer et al., 2020).

Original: However, this approach has some limitations: i) it is challenging to interpret how the model is misspecified – an obvious choice is to use posterior predictive checks, but poor predictive performance could result from failed inference, limitations of simulator, or both; ii) retraining of the approximate posterior would be required to investigate different error models, which can be computationally expensive; and iii) the error model itself is likely tractable, which can be utilised to improve inference.
Revised: However, this approach has some limitations: i) it is challenging to interpret how the model is misspecified – an obvious choice is to use posterior predictive checks, but poor predictive performance could result from failed inference, limitations of simulator, or both; ii) retraining of the approximate posterior would be required to investigate different error models, which can be computationally expensive; and iii) the error model itself is likely tractable, which can be utilised to improve inference.

Original: Our contribution.
Revised: Our contribution.

Original: We propose robust neural posterior estimation ( RNPE ), an SBI framework that incorporates an error model, p ( y | x ) to account for discrepancies between simulations and the observed data, whilst avoiding the limitations of naïvely incorporating it directly into the simulator.
Revised: We propose robust neural posterior estimation ( RNPE ), an SBI framework that incorporates an error model, p ( y | x ) to account for discrepancies between simulations and the observed data, whilst avoiding the limitations of naïvely incorporating it directly into the simulator.

Original: We explicitly learn to invert the error model for the observed data prior to inference of the parameter posterior, a process we call ‘denoising’.
Revised: We explicitly learn to invert the error model for the observed data prior to inference of the parameter posterior, a process we call ‘denoising’.

Original: This approach yields the following advantages:
Revised: This approach yields the following advantages:

Original: Efficient Model Criticism.
Revised: Efficient Model Criticism.

Original: Understanding a model’s limitations is crucial for principled model development (Box, 1980; Gelman and Shalizi, 2013).
Revised: Understanding a model’s limitations is crucial for principled model development (Box, 1980; Gelman and Shalizi, 2013).

Original: By probabilistically inverting the error process for the observed data, and inferring any associated error model parameters, we can detect and interpret discrepancies between the simulated and observed data, allowing criticism of the simulator.
Revised: By probabilistically inverting the error process for the observed data, and inferring any associated error model parameters, we can detect and interpret discrepancies between the simulated and observed data, allowing criticism of the simulator.

Original: This process can be achieved in a manner decoupled from inference, which provides efficiency benefits, whilst avoiding confounding inference failures with problems arising due to limitations of the simulator.
Revised: This process can be achieved in a manner decoupled from inference, which provides efficiency benefits, whilst avoiding confounding inference failures with problems arising due to limitations of the simulator.

Original: Robust Inference.
Revised: Robust Inference.

Original: Even after several rounds of model development and model criticism, the best available model will still frequently be misspecified to an appreciable degree.
Revised: Even after several rounds of model development and model criticism, the best available model will still frequently be misspecified to an appreciable degree.

Original: However, it is often still desirable to use ‘wrong but useful’ models for parameter inference and predictions, and hence the inference process should be robust to misspecification.
Revised: However, it is often still desirable to use ‘wrong but useful’ models for parameter inference and predictions, and hence the inference process should be robust to misspecification.

Original: By exploiting the amortisationproperty of NPE , we can ensemble posterior estimates over a set of ‘denoised’ observations, leading to posterior estimates robust to misspecification.
Revised: By exploiting the amortisationproperty of NPE , we can ensemble posterior estimates over a set of ‘denoised’ observations, leading to posterior estimates robust to misspecification.

Original: An overview of the RNPE framework can be seen in Fig.
Revised: An overview of the RNPE framework can be seen in Fig.

Original: We examine the practical value of this approach for model criticism and robust inference on examples in which we artificially introduce realistic levels of misspecification.
Revised: We examine the practical value of this approach for model criticism and robust inference on examples in which we artificially introduce realistic levels of misspecification.

Original:  Method 2.1 Error Model
Revised:  Method 2.1 Error Model

Original: As outlined above, we can explicitly model discrepancies between the observed data and simulations by introducing an error model p ( y | x ) , such that the assumed generative model is
Revised: As outlined above, we can explicitly model discrepancies between the observed data and simulations by introducing an error model p ( y | x ) , such that the assumed generative model is

Original: [Equation]
Revised: [Equation]

Original: where we assume that the error is independent of the simulator parameters θ , and we treat x as an unobserved latent variable of the model.
Revised: where we assume that the error is independent of the simulator parameters θ , and we treat x as an unobserved latent variable of the model.

Original: Using the equivalent factorisation
Revised: Using the equivalent factorisation

Original: [Equation]
Revised: [Equation]

Original: we can marginalise over the latent x and find an expression for the parameter posterior
Revised: we can marginalise over the latent x and find an expression for the parameter posterior

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Equation (3), implies that if we had access to p ( x | y ) and p ( θ | x ) , we could sample from the parameter posterior by first sampling from the posterior over the latent variables x ∼ p ( x | y ) , and then sampling from p ( θ | x ) .
Revised: Equation (3), implies that if we had access to p ( x | y ) and p ( θ | x ) , we could sample from the parameter posterior by first sampling from the posterior over the latent variables x ∼ p ( x | y ) , and then sampling from p ( θ | x ) .

Original: This approach reverses the data generating process from Equation (1), whilst propagating uncertainty throughout, first removing the error from the observed data by denoising, and then finding the associated simulator parameters given the denoised data.
Revised: This approach reverses the data generating process from Equation (1), whilst propagating uncertainty throughout, first removing the error from the observed data by denoising, and then finding the associated simulator parameters given the denoised data.

Original: A Monte
Revised: A Monte

Original: Carlo approximation of the expectation in Equation (3) can also be used to estimate the posterior density p ( θ | y ) , i.e.
Revised: Carlo approximation of the expectation in Equation (3) can also be used to estimate the posterior density p ( θ | y ) , i.e.

Original: [Equation]
Revised: [Equation]

Original: however, we note that in some scenarios this approximation could have high variance.
Revised: however, we note that in some scenarios this approximation could have high variance.

Original: In practice, p ( x | y ) and p ( θ | x ) are unknown.
Revised: In practice, p ( x | y ) and p ( θ | x ) are unknown.

Original: We can approximate p ( θ | x ) by training a normalising flow q ( θ | x ) on simulated data ( x , θ ) ∼ p ( x | θ ) p ( θ ) , as is commonly done for NPE .
Revised: We can approximate p ( θ | x ) by training a normalising flow q ( θ | x ) on simulated data ( x , θ ) ∼ p ( x | θ ) p ( θ ) , as is commonly done for NPE .

Original: To approximately sample from p ( x | y ) , we i) specify an error model p ( y | x ) , ii) train a normalising flow, q ( x ) , fitted to samples from the prior predictive distribution of the simulator x ∼ p ( x ) , and iii) sample ˜ x ∼ p ( x | y ) , along with any error model parameters, using MCMC , as ˆ p ( x | y )
Revised: To approximately sample from p ( x | y ) , we i) specify an error model p ( y | x ) , ii) train a normalising flow, q ( x ) , fitted to samples from the prior predictive distribution of the simulator x ∼ p ( x ) , and iii) sample ˜ x ∼ p ( x | y ) , along with any error model parameters, using MCMC , as ˆ p ( x | y )

Original: ∝ p ( y | x ) q ( x ) .
Revised: ∝ p ( y | x ) q ( x ) .

Original: We denote samples from ˆ p ( x | y ) and q ( θ | x ) as ˜ x and ˜ θ to avoid confusion with prior samples and simulations.
Revised: We denote samples from ˆ p ( x | y ) and q ( θ | x ) as ˜ x and ˜ θ to avoid confusion with prior samples and simulations.

Original: Pseudo-code for the overall approach is given in Algorithm 1.
Revised: Pseudo-code for the overall approach is given in Algorithm 1.

Original: MCMC
Revised: MCMC

Original: , M
Revised: , M

Original: Under this framework, the standard SBI approach can be retrieved as the special case in which p ( y | x ) = p ( x | y )
Revised: Under this framework, the standard SBI approach can be retrieved as the special case in which p ( y | x ) = p ( x | y )

Original: =
Revised: =

Original: δ ( x − y ) , where δ is the Dirac delta distribution, meaning that we revert to assuming no discrepancy between the simulator and data generating process.
Revised: δ ( x − y ) , where δ is the Dirac delta distribution, meaning that we revert to assuming no discrepancy between the simulator and data generating process.

Original: The idealised nature of simulation models implies that frequently there will be some characteristics of the observed data that are well captured by simulations, and some aspects in which there exists a discrepancy.
Revised: The idealised nature of simulation models implies that frequently there will be some characteristics of the observed data that are well captured by simulations, and some aspects in which there exists a discrepancy.

Original: Using a set of summary statistics for inference, rather than the raw data naturally captures an interpretable, low dimensional and diverse set of characteristics of the simulated and observed data.
Revised: Using a set of summary statistics for inference, rather than the raw data naturally captures an interpretable, low dimensional and diverse set of characteristics of the simulated and observed data.

Original: Because of this, we find that in general, reducing the raw data to summary statistics is advantageous for model criticism , as the practitioner can choose summary statistics which align with their belief about important model characteristics, and easily interpret any discrepancies (see Section 5).
Revised: Because of this, we find that in general, reducing the raw data to summary statistics is advantageous for model criticism , as the practitioner can choose summary statistics which align with their belief about important model characteristics, and easily interpret any discrepancies (see Section 5).

Original: The requirement for domain specific knowledge to develop summary statistics has been alleviated by recent work (e.g. Fearnhead and Prangle, 2012; Chan et al., 2018; Chen et al., 2020), which seek to automatically embed the raw data to summary statistics; however, these embeddings often lack a tangible interpretation.
Revised: The requirement for domain specific knowledge to develop summary statistics has been alleviated by recent work (e.g. Fearnhead and Prangle, 2012; Chan et al., 2018; Chen et al., 2020), which seek to automatically embed the raw data to summary statistics; however, these embeddings often lack a tangible interpretation.

Original: Additionally, the embedding methods themselves may not be robust to misspecification.
Revised: Additionally, the embedding methods themselves may not be robust to misspecification.

Original: One approach is to use both hand-crafted and automatic summary statistics in tandem.
Revised: One approach is to use both hand-crafted and automatic summary statistics in tandem.

Original: Hereafter, however, we generally perform model criticism and inference using hand-crafted summary statistics, such that x and y o refer to the simulated and observed summary statistics, not the raw data.
Revised: Hereafter, however, we generally perform model criticism and inference using hand-crafted summary statistics, such that x and y o refer to the simulated and observed summary statistics, not the raw data.

Original: 2.2 Spike and Slab
Revised: 2.2 Spike and Slab

Original: In practice, the true error model is unknown, and hence the error model should describe our prior belief over the discrepancy between the data generating process and simulations.
Revised: In practice, the true error model is unknown, and hence the error model should describe our prior belief over the discrepancy between the data generating process and simulations.

Original: Additionally, to facilitate model criticism, it should ideally allow easy assessment of which summary statistics are approximately well-specified and which are misspecified.
Revised: Additionally, to facilitate model criticism, it should ideally allow easy assessment of which summary statistics are approximately well-specified and which are misspecified.

Original: Inspired by sparsity-inducing priors used in Bayesian regression literature (George and McCulloch, 1993), we use a “spike and slab” error model on each summary statistic
Revised: Inspired by sparsity-inducing priors used in Bayesian regression literature (George and McCulloch, 1993), we use a “spike and slab” error model on each summary statistic

Original: [Equation]
Revised: [Equation]

Original: where x ∈ R D .
Revised: where x ∈ R D .

Original: As the summary statistics have varying scales, we standardise them to have mean zero and variance one.
Revised: As the summary statistics have varying scales, we standardise them to have mean zero and variance one.

Original: The Bernoulli variables indicate whether the j -th summary statistic is approximately well-specified; we use the probability of ρ = 0 .
Revised: The Bernoulli variables indicate whether the j -th summary statistic is approximately well-specified; we use the probability of ρ = 0 .

Original: 5 to express the belief that it is equally likely a summary statistic will be misspecified or well-specified a priori .
Revised: 5 to express the belief that it is equally likely a summary statistic will be misspecified or well-specified a priori .

Original: When a summary statistic is well-specified (i.e.,
Revised: When a summary statistic is well-specified (i.e.,

Original: z j = 0 ), we take the error distribution to be a tight Gaussian (a spike) centred on x j , choosing σ = 0 .
Revised: z j = 0 ), we take the error distribution to be a tight Gaussian (a spike) centred on x j , choosing σ = 0 .

Original: 01 to enforce consistency between the simulator and the observed data for the j -th statistic.
Revised: 01 to enforce consistency between the simulator and the observed data for the j -th statistic.

Original: In contrast, in the misspecified case (i.e., z j = 1 ), we take the error model to be the much wider and heavy tailed Cauchy distribution (a slab).
Revised: In contrast, in the misspecified case (i.e., z j = 1 ), we take the error model to be the much wider and heavy tailed Cauchy distribution (a slab).

Original: Generally, we might expect misspecification to be relatively subtle, but some model inadequacies can lead to catastrophically misspecified summary statistics.
Revised: Generally, we might expect misspecification to be relatively subtle, but some model inadequacies can lead to catastrophically misspecified summary statistics.

Original: We chose the Cauchy scale τ = 0 .
Revised: We chose the Cauchy scale τ = 0 .

Original: 25 , to reflect this, as it places half the mass within ± 0 .
Revised: 25 , to reflect this, as it places half the mass within ± 0 .

Original: 25 standard deviations of x j , but the long tails accommodate summary statistics that are highly misspecified.
Revised: 25 standard deviations of x j , but the long tails accommodate summary statistics that are highly misspecified.

Original: The sparsity-inducing effect of this error model matches what we consider to be a common scenario, namely that a proportion of the summary statistics are jointly consistent with the simulator, whereas others may be incompatible.
Revised: The sparsity-inducing effect of this error model matches what we consider to be a common scenario, namely that a proportion of the summary statistics are jointly consistent with the simulator, whereas others may be incompatible.

Original: If we expect a priori more or fewer of the summary statistics to be misspecified, the prior misspecification probability can be adjusted accordingly.
Revised: If we expect a priori more or fewer of the summary statistics to be misspecified, the prior misspecification probability can be adjusted accordingly.

Original: By marginalising over z , the spike and slab error model can also be written as
Revised: By marginalising over z , the spike and slab error model can also be written as

Original: [Equation]
Revised: [Equation]

Original: and as such can be viewed as an equally weighted mixture of the Gaussian spike, p ( y j | x j , z j = 0) , and the Cauchy slab, p ( y j | x j , z j = 1) , for each summary statistic.
Revised: and as such can be viewed as an equally weighted mixture of the Gaussian spike, p ( y j | x j , z j = 0) , and the Cauchy slab, p ( y j | x j , z j = 1) , for each summary statistic.

Original: We note that error model hyperparameter tuning approaches could be considered, e.g. a reasonable heuristic would be to choose an error model that is broad enough that the denoised data ˜ x tend not to be outliers in q ( x ) , compared to simulations 2 .
Revised: We note that error model hyperparameter tuning approaches could be considered, e.g. a reasonable heuristic would be to choose an error model that is broad enough that the denoised data ˜ x tend not to be outliers in q ( x ) , compared to simulations 2 .

Original: However, we chose to keep the hyperparameters consistent across tasks, to avoid the risk of overfitting to the tasks and to demonstrate that neither strong prior knowledge on the error model, nor careful hyperparameter tuning is necessary to yield substantial improvements in performance.
Revised: However, we chose to keep the hyperparameters consistent across tasks, to avoid the risk of overfitting to the tasks and to demonstrate that neither strong prior knowledge on the error model, nor careful hyperparameter tuning is necessary to yield substantial improvements in performance.

Original: A key advantage of the spike and slab error model is given by the latent variable z .
Revised: A key advantage of the spike and slab error model is given by the latent variable z .

Original: Similar to posterior inclusion probabilities in Bayesian regression, the posterior frequency of being in the slab,
Revised: Similar to posterior inclusion probabilities in Bayesian regression, the posterior frequency of being in the slab,

Original: Pr ( z j = 1 | y ) , can be used as an indicator of the posterior misspecification probability for the j -th summary statistic.
Revised: Pr ( z j = 1 | y ) , can be used as an indicator of the posterior misspecification probability for the j -th summary statistic.

Original: By comparing to the prior misspecification probability ρ , we can say that if
Revised: By comparing to the prior misspecification probability ρ , we can say that if

Original: Pr ( z j = 1 | y ) > ρ , the model provides evidence of misspecification for the j -th summary statistic, and if Pr ( z j = 1 | y ) <
Revised: Pr ( z j = 1 | y ) > ρ , the model provides evidence of misspecification for the j -th summary statistic, and if Pr ( z j = 1 | y ) <

Original: ρ , it provides evidence it is well-specified (Talbott, 2016).
Revised: ρ , it provides evidence it is well-specified (Talbott, 2016).

Original: For the purpose of generality, the latent variable z was not included when describing RNPE thus far.
Revised: For the purpose of generality, the latent variable z was not included when describing RNPE thus far.

Original: However, we can jointly infer the posterior ˆ p ( x , z | y ) in step 5 of algorithm 1, by using an MCMC algorithm that supports sampling both continuous and discrete variables.
Revised: However, we can jointly infer the posterior ˆ p ( x , z | y ) in step 5 of algorithm 1, by using an MCMC algorithm that supports sampling both continuous and discrete variables.

Original: We used mixed Hamiltonian Monte
Revised: We used mixed Hamiltonian Monte

Original: Carlo ( HMC ) (Zhou, 2020, 2022) from the NumPyro python package (Phan et al., 2019), which is an adaptation of HMC (Neal et al., 2011; Duane et al., 1987), for this purpose.
Revised: Carlo ( HMC ) (Zhou, 2020, 2022) from the NumPyro python package (Phan et al., 2019), which is an adaptation of HMC (Neal et al., 2011; Duane et al., 1987), for this purpose.

Original:  Related Work 3.1 Model Criticism in SBI
Revised:  Related Work 3.1 Model Criticism in SBI

Original: Posterior predictive checks have been widely used in SBI , to compare the predictive samples to the observed data (e.g. Durkan et al., 2020; Greenberg et al., 2019; Papamakarios et al., 2019b).
Revised: Posterior predictive checks have been widely used in SBI , to compare the predictive samples to the observed data (e.g. Durkan et al., 2020; Greenberg et al., 2019; Papamakarios et al., 2019b).

Original: Although this is a form of model criticism, we note a key limitation of this approach is that if a discrepancy is discovered, it may not be clear whether this is due to the failure of the inference procedure, or due to simulator misspecification.
Revised: Although this is a form of model criticism, we note a key limitation of this approach is that if a discrepancy is discovered, it may not be clear whether this is due to the failure of the inference procedure, or due to simulator misspecification.

Original: In general, it may be possible to identify the presence of misspecification using anomaly/novelty detection.
Revised: In general, it may be possible to identify the presence of misspecification using anomaly/novelty detection.

Original: One such approach was suggested by Schmitt et al.
Revised: One such approach was suggested by Schmitt et al.

Original: (2021) in the context of SBI .
Revised: (2021) in the context of SBI .

Original: However, this approach does not facilitate interpreting misspecification in the sense presented here, and it is not clear how to extend it to perform robust inference.
Revised: However, this approach does not facilitate interpreting misspecification in the sense presented here, and it is not clear how to extend it to perform robust inference.

Original: There are a few related methods that criticise a posterior estimate by assessing its calibration using simulated data (e.g. Hermans et al., 2021; Prangle et al., 2014; Talts et al., 2018); however, these approaches criticise the inference procedure, not the simulator itself, and the associated results rely on the simulator being well-specified.
Revised: There are a few related methods that criticise a posterior estimate by assessing its calibration using simulated data (e.g. Hermans et al., 2021; Prangle et al., 2014; Talts et al., 2018); however, these approaches criticise the inference procedure, not the simulator itself, and the associated results rely on the simulator being well-specified.

Original: 3.2 Robust Inference in SBI
Revised: 3.2 Robust Inference in SBI

Original: Approximate Bayesian computation ( ABC ) is a family of SBI methods characterised by their use of rejection sampling alongside a kernel function to compare the simulated and observed data (Tavaré et al., 1997; Pritchard et al., 1999;
Revised: Approximate Bayesian computation ( ABC ) is a family of SBI methods characterised by their use of rejection sampling alongside a kernel function to compare the simulated and observed data (Tavaré et al., 1997; Pritchard et al., 1999;

Original: Beaumont et al., 2002).
Revised: Beaumont et al., 2002).

Original: The simplest form of ABC samples candidate parameters from the prior distribution and accepts them with probability proportional to the similarity of the corresponding simulations to the observed data.
Revised: The simplest form of ABC samples candidate parameters from the prior distribution and accepts them with probability proportional to the similarity of the corresponding simulations to the observed data.

Original: The use of a kernel is often seen as a necessary evil, as enforcing an exact match between the observed and simulated data generally results in all samples being rejected.
Revised: The use of a kernel is often seen as a necessary evil, as enforcing an exact match between the observed and simulated data generally results in all samples being rejected.

Original: However, it has been noted that the kernel function can often be interpreted as specifying an error model distribution p ( y | x ) (Wilkinson, 2013).
Revised: However, it has been noted that the kernel function can often be interpreted as specifying an error model distribution p ( y | x ) (Wilkinson, 2013).

Original: This idea has been expanded using generalised Bayesian inference, in which the kernel is replaced with a loss function, which can be chosen to use a robust measure of discrepancy such as the maximum mean discrepancy ( MMD ) (Chérief-Abdellatif and Alquier, 2020;
Revised: This idea has been expanded using generalised Bayesian inference, in which the kernel is replaced with a loss function, which can be chosen to use a robust measure of discrepancy such as the maximum mean discrepancy ( MMD ) (Chérief-Abdellatif and Alquier, 2020;

Original: Frazier et al., 2020a; Fujisawa et al., 2021;
Revised: Frazier et al., 2020a; Fujisawa et al., 2021;

Original: Park et al., 2015; Schmon et al., 2020; Dyer et al., 2021).
Revised: Park et al., 2015; Schmon et al., 2020; Dyer et al., 2021).

Original: Synthetic likelihood is an SBI method that uses a Gaussian approximation of the likelihood (Wood, 2010).
Revised: Synthetic likelihood is an SBI method that uses a Gaussian approximation of the likelihood (Wood, 2010).

Original: Frazier and Drovandi (2021) investigated a robust synthetic likelihood algorithm, in which the variance components of the Gaussian likelihood are expanded based on additional free parameters, inferred in a nested MCMC sampling scheme.
Revised: Frazier and Drovandi (2021) investigated a robust synthetic likelihood algorithm, in which the variance components of the Gaussian likelihood are expanded based on additional free parameters, inferred in a nested MCMC sampling scheme.

Original: In the context of the paper, they interpret this approach as “likelihood adjustment”; however, this could equivalently be reframed as the addition of an independent Gaussian error model, with a prior on the variances 3 .
Revised: In the context of the paper, they interpret this approach as “likelihood adjustment”; however, this could equivalently be reframed as the addition of an independent Gaussian error model, with a prior on the variances 3 .

Original: In contrast to more traditional SBI methods, few solutions have been proposed for performing robust inference with neural SBI methods.
Revised: In contrast to more traditional SBI methods, few solutions have been proposed for performing robust inference with neural SBI methods.

Original: Perhaps the most common approach is to augment the model with additional nuisance parameters to expand the range of values the simulator can reasonably produce (Cranmer et al., 2020).
Revised: Perhaps the most common approach is to augment the model with additional nuisance parameters to expand the range of values the simulator can reasonably produce (Cranmer et al., 2020).

Original: This again can often be interpreted as a method of incorporating an error model within the simulator.
Revised: This again can often be interpreted as a method of incorporating an error model within the simulator.

Original: However, as previously discussed, the primary limitation of this approach is that it does not easily facilitate model criticism .
Revised: However, as previously discussed, the primary limitation of this approach is that it does not easily facilitate model criticism .

Original:  Experimental setup
Revised:  Experimental setup

Original: On three misspecified tasks, we benchmark the performance of three methods for parameter inference: i) RNPE , ii) noisy neural posterior estimation ( NNPE ) in which we directly approximate p ( θ | y ) by incorporating the error model into the simulator, and iii) NPE , in which we assume no model error is present.
Revised: On three misspecified tasks, we benchmark the performance of three methods for parameter inference: i) RNPE , ii) noisy neural posterior estimation ( NNPE ) in which we directly approximate p ( θ | y ) by incorporating the error model into the simulator, and iii) NPE , in which we assume no model error is present.

Original: To reliably assess performance, we repeat the inference procedure with 1000 different observations and ground truth parameter pairs for each misspecified task, and calculate two metrics: i) the log probability of the true parameter vector, θ ∗ , and ii) the posterior coverage properties (see
Revised: To reliably assess performance, we repeat the inference procedure with 1000 different observations and ground truth parameter pairs for each misspecified task, and calculate two metrics: i) the log probability of the true parameter vector, θ ∗ , and ii) the posterior coverage properties (see

Original: Section 4.2).
Revised: Section 4.2).

Original: Further metrics, and results for the well-specified case, can be found in Appendix A.
Revised: Further metrics, and results for the well-specified case, can be found in Appendix A.

Original: We further show examples of RNPE on a single observation for each task to demonstrate how RNPE can be applied in practice for model criticism and robust posterior inference.
Revised: We further show examples of RNPE on a single observation for each task to demonstrate how RNPE can be applied in practice for model criticism and robust posterior inference.

Original: For all experiments, we used N = 50 , 000 simulations, with M = 100 , 000 MCMC samples following 20 , 000 warm up steps.
Revised: For all experiments, we used N = 50 , 000 simulations, with M = 100 , 000 MCMC samples following 20 , 000 warm up steps.

Original: The MCMC chains were initialised using a random simulation, and z j = 1 for j = 1 , . . .
Revised: The MCMC chains were initialised using a random simulation, and z j = 1 for j = 1 , . . .

Original: , D .
Revised: , D .

Original: To build the approximation q ( x ) , we used block neural autoregressive flows (De Cao et al., 2020), as we do not require the ability to sample directly from q ( x ) .
Revised: To build the approximation q ( x ) , we used block neural autoregressive flows (De Cao et al., 2020), as we do not require the ability to sample directly from q ( x ) .

Original: For the approximation of q ( θ | x ) , we used neural spline flows (Durkan et al., 2019).
Revised: For the approximation of q ( θ | x ) , we used neural spline flows (Durkan et al., 2019).

Original: For all tasks the hyperparameters were kept consistent; information on hyperparameter choices can be found in
Revised: For all tasks the hyperparameters were kept consistent; information on hyperparameter choices can be found in

Original: Appendix C.
Revised: Appendix C.

Original: The code required to reproduce all the results from this manuscript is available at https://github
Revised: The code required to reproduce all the results from this manuscript is available at https://github

Original: 4.1 Tasks
Revised: 4.1 Tasks

Original: Detailed descriptions of each task can be found in Appendix B.
Revised: Detailed descriptions of each task can be found in Appendix B.

Original: Gaussian.
Revised: Gaussian.

Original: A simple Gaussian example from Frazier et al.
Revised: A simple Gaussian example from Frazier et al.

Original: (2020b).
Revised: (2020b).

Original: The task involves predicting a single parameter µ , using the sample mean and variance of 100 i.i.d.
Revised: The task involves predicting a single parameter µ , using the sample mean and variance of 100 i.i.d.

Original: samples drawn from N ( µ, 1) , where 1 is the variance of the Gaussian distribution.
Revised: samples drawn from N ( µ, 1) , where 1 is the variance of the Gaussian distribution.

Original: To artificially produce misspecified observations, the 100 i.i.d.
Revised: To artificially produce misspecified observations, the 100 i.i.d.

Original: are instead drawn from N ( µ, 2) .
Revised: are instead drawn from N ( µ, 2) .

Original: Despite its simplicity, this task usefully demonstrates that issues with misspecification can arise even in the simplest of models.
Revised: Despite its simplicity, this task usefully demonstrates that issues with misspecification can arise even in the simplest of models.

Original: Gaussian Linear.
Revised: Gaussian Linear.

Original: A 10-d linear Gaussian model from Lueckmann et al.
Revised: A 10-d linear Gaussian model from Lueckmann et al.

Original: The parameter θ is the mean vector of a Gaussian likelihood.
Revised: The parameter θ is the mean vector of a Gaussian likelihood.

Original: Additive Gaussian noise is used to introduce misspecification.
Revised: Additive Gaussian noise is used to introduce misspecification.

Original: For this task, the spike and slab error model employed will be reasonably misspecified, as the noise is not sparse or heavy tailed.
Revised: For this task, the spike and slab error model employed will be reasonably misspecified, as the noise is not sparse or heavy tailed.

Original: SIR.
Revised: SIR.

Original: A stochastic model of epidemic spread with a time-varying infection rate.
Revised: A stochastic model of epidemic spread with a time-varying infection rate.

Original: We attempt to infer two parameters, the infection rate β and the recovery rate γ .
Revised: We attempt to infer two parameters, the infection rate β and the recovery rate γ .

Original: We computed the following summary statistics: 1-3) Mean , Median and Max - the mean, median and maximum number of infections; 4)
Revised: We computed the following summary statistics: 1-3) Mean , Median and Max - the mean, median and maximum number of infections; 4)

Original: Max Day - the day of occurrence of the maximum number of infections, 5) Half Day - the day at which half the total number of infections was reached, and 6) Autocor - the mean autocorrelation (lag 1) of infections.
Revised: Max Day - the day of occurrence of the maximum number of infections, 5) Half Day - the day at which half the total number of infections was reached, and 6) Autocor - the mean autocorrelation (lag 1) of infections.

Original: To artificially produce misspecified observations, we chose to introduce a small degree of reporting delays, whereby weekend infection counts are reduced by 5%, which are subsequently recouped on the following Monday.
Revised: To artificially produce misspecified observations, we chose to introduce a small degree of reporting delays, whereby weekend infection counts are reduced by 5%, which are subsequently recouped on the following Monday.

Original: Fig.
Revised: Fig.

Original: 4d shows an example of a raw observation.
Revised: 4d shows an example of a raw observation.

Original: CS.
Revised: CS.

Original: A marked point process model of cancer and stromal (CS) cell development in 2D-space, in which the locality of a cell relative to unobserved parent points determines whether it is a cancer or a stromal cell.
Revised: A marked point process model of cancer and stromal (CS) cell development in 2D-space, in which the locality of a cell relative to unobserved parent points determines whether it is a cancer or a stromal cell.

Original: Similar point process models can be found in Jones-Todd et al.
Revised: Similar point process models can be found in Jones-Todd et al.

Original: There are three
Revised: There are three

Original: Poisson rate parameters, corresponding to the cell rate λ c , the parent rate λ p and the daughter rate λ d , where λ d controls the number of nearest cells to each parent which become cancerous.
Revised: Poisson rate parameters, corresponding to the cell rate λ c , the parent rate λ p and the daughter rate λ d , where λ d controls the number of nearest cells to each parent which become cancerous.

Original: The following summary statistics were used: 1-2) N Cancer and N Stromal - the number of cancer and stromal cells; 3-4) Mean Min Dist and Max Min Dist - the mean and maximum distance from stromal cells to their nearest cancer cell.
Revised: The following summary statistics were used: 1-2) N Cancer and N Stromal - the number of cancer and stromal cells; 3-4) Mean Min Dist and Max Min Dist - the mean and maximum distance from stromal cells to their nearest cancer cell.

Original: To artificially produce misspecified observations, cancer cells were removed if they fell too close to a parent point, mimicking necrosis that often occurs in core regions of tumours.
Revised: To artificially produce misspecified observations, cancer cells were removed if they fell too close to a parent point, mimicking necrosis that often occurs in core regions of tumours.

Original: A plot of an example misspecified simulation and a corresponding artificially produced observation can be found in Fig.
Revised: A plot of an example misspecified simulation and a corresponding artificially produced observation can be found in Fig.

Original: 9 in the appendix.
Revised: 9 in the appendix.

Original: 4.2 Performance Metrics
Revised: 4.2 Performance Metrics

Original: Log probability of θ ∗ .
Revised: Log probability of θ ∗ .

Original: The mean posterior log probability of the true parameters over multiple observations is an extensively used performance metric (e.g. Papamakarios and Murray, 2016;
Revised: The mean posterior log probability of the true parameters over multiple observations is an extensively used performance metric (e.g. Papamakarios and Murray, 2016;

Original: Hermans et al., 2020; Durkan et al., 2020), and is closely related to the average KL divergence between the true and approximate posteriors (Lueckmann et al., 2021).
Revised: Hermans et al., 2020; Durkan et al., 2020), and is closely related to the average KL divergence between the true and approximate posteriors (Lueckmann et al., 2021).

Original: We found that NPE would occasionally fail catastrophically, placing negligible posterior mass on the true parameters, skewing the mean estimates.
Revised: We found that NPE would occasionally fail catastrophically, placing negligible posterior mass on the true parameters, skewing the mean estimates.

Original: Due to this, we instead use box plots to visualise the distribution over the log-probabilities of the true parameters, rather than reporting the mean.
Revised: Due to this, we instead use box plots to visualise the distribution over the log-probabilities of the true parameters, rather than reporting the mean.

Original: Posterior Coverage.
Revised: Posterior Coverage.

Original: Given a confidence level α and a posterior approximation ˆ p ( θ | y ) , let
Revised: Given a confidence level α and a posterior approximation ˆ p ( θ | y ) , let

Original: HDR ˆ p ( θ | y ) (1 − α ) represent its 1 − α highest posterior density region, i.e. the smallest region that contains at least 100(1 − α )% of the mass of ˆ p ( θ | y ) .
Revised: HDR ˆ p ( θ | y ) (1 − α ) represent its 1 − α highest posterior density region, i.e. the smallest region that contains at least 100(1 − α )% of the mass of ˆ p ( θ | y ) .

Original: The expected posterior coverage is the frequency with which the true parameter value falls within this highest density region
Revised: The expected posterior coverage is the frequency with which the true parameter value falls within this highest density region

Original: [Equation]
Revised: [Equation]

Original: (9) where 1 is the indicator function.
Revised: (9) where 1 is the indicator function.

Original: Posterior coverage is useful for assessing whether an inference procedure is likely to yield overconfident or conservative posterior estimates (Hermans et al., 2021).
Revised: Posterior coverage is useful for assessing whether an inference procedure is likely to yield overconfident or conservative posterior estimates (Hermans et al., 2021).

Original: We empirically estimate this expectation, approximating the highest density regions using the density quantile approach outlined in Hyndman (1996), with 10,000 samples from the approximate posterior.
Revised: We empirically estimate this expectation, approximating the highest density regions using the density quantile approach outlined in Hyndman (1996), with 10,000 samples from the approximate posterior.

Original:  Results and Discussion 5.1 Overall Performance
Revised:  Results and Discussion 5.1 Overall Performance

Original: The results shown in Fig.
Revised: The results shown in Fig.

Original: 2 show that both RNPE and NNPE perform better than NPE , both in terms of producing posteriors with more conservative coverage properties (Fig. 2a) and by tending to place more mass on θ ∗ (Fig. 2b).
Revised: 2 show that both RNPE and NNPE perform better than NPE , both in terms of producing posteriors with more conservative coverage properties (Fig. 2a) and by tending to place more mass on θ ∗ (Fig. 2b).

Original: It is important to note that having conservative posterior estimates is generally considered preferable to having overconfident posterior estimates, as the latter could lead to drawing of erroneous scientific conclusions.
Revised: It is important to note that having conservative posterior estimates is generally considered preferable to having overconfident posterior estimates, as the latter could lead to drawing of erroneous scientific conclusions.

Original: R NPE slightly outperforms NNPE in terms of the mass placed on θ ∗ .
Revised: R NPE slightly outperforms NNPE in terms of the mass placed on θ ∗ .

Original: We hypothesise this increased performance is due to RNPE exploiting the fact that the
Revised: We hypothesise this increased performance is due to RNPE exploiting the fact that the

Original: E m p
Revised: E m p

Original: i r i ca l
Revised: i r i ca l

Original: C ov e r a g e (a)
Revised: C ov e r a g e (a)

Original: Gaussian 0.
Revised: Gaussian 0.

Original: Gaussian Linear 0.
Revised: Gaussian Linear 0.

Original: SIR 0.
Revised: SIR 0.

Original: CS
Revised: CS

Original: L og
Revised: L og

Original: P r ob a b ilit y * (b)
Revised: P r ob a b ilit y * (b)

Original: Gaussian
Revised: Gaussian

Original: Gaussian Linear
Revised: Gaussian Linear

Original: SIR
Revised: SIR

Original: CS
Revised: CS

Original: Confidence Level
Revised: Confidence Level

Original: RNPE
Revised: RNPE

Original: NNPE
Revised: NNPE

Original: NPE error model is known and tractable, rather than naïvely relying on NPE to account for it.
Revised: NPE error model is known and tractable, rather than naïvely relying on NPE to account for it.

Original: For all the tasks the error model is misspecified: these results highlight the importance of accounting for model misspecification, even if doing so may introduce a misspecified error model.
Revised: For all the tasks the error model is misspecified: these results highlight the importance of accounting for model misspecification, even if doing so may introduce a misspecified error model.

Original: 5.2 Examples
Revised: 5.2 Examples

Original: Gaussian.
Revised: Gaussian.

Original: An example for the Gaussian task can be seen in Fig.
Revised: An example for the Gaussian task can be seen in Fig.

Original: 1, in which x 1 corresponds to the mean statistic, and x 2 the variance.
Revised: 1, in which x 1 corresponds to the mean statistic, and x 2 the variance.

Original: From observing the posterior misspecification probabilities and the denoised samples, the results clearly identify a discrepancy in which the observed variance statistic is higher than the simulated variance statistics, allowing identification of source of misspecification.
Revised: From observing the posterior misspecification probabilities and the denoised samples, the results clearly identify a discrepancy in which the observed variance statistic is higher than the simulated variance statistics, allowing identification of source of misspecification.

Original: As this is a simple example, we can analytically access the true posterior (obtained using the true data generating model).
Revised: As this is a simple example, we can analytically access the true posterior (obtained using the true data generating model).

Original: R NPE produces a posterior close to the ground-truth, whereas NPE confidently places most of the posterior mass far from the ground-truth posterior.
Revised: R NPE produces a posterior close to the ground-truth, whereas NPE confidently places most of the posterior mass far from the ground-truth posterior.

Original: [Figure]
Revised: [Figure]

Original: example of the Gaussian Linear task.
Revised: example of the Gaussian Linear task.

Original: Gaussian Linear.
Revised: Gaussian Linear.

Original: For the Gaussian Linear task, misspecification was introduced by applying additive Gaussian noise.
Revised: For the Gaussian Linear task, misspecification was introduced by applying additive Gaussian noise.

Original: Due to the lack of structure (e.g. correlations between simulation output dimensions), and the lack of sparsity in the errors, the model generally cannot provide much evidence for or against misspecification, as shown by Fig. 3.
Revised: Due to the lack of structure (e.g. correlations between simulation output dimensions), and the lack of sparsity in the errors, the model generally cannot provide much evidence for or against misspecification, as shown by Fig. 3.

Original: Corresponding pair plots of the denoised data and posterior are shown in
Revised: Corresponding pair plots of the denoised data and posterior are shown in

Original: Figs.
Revised: Figs.

Original: 10-11, in the appendix.
Revised: 10-11, in the appendix.

Original: SIR.
Revised: SIR.

Original: In the SIR model, the observed data were artificially corrupted by mimicking reporting delays over the weekend.
Revised: In the SIR model, the observed data were artificially corrupted by mimicking reporting delays over the weekend.

Original: With this in mind, we can consider how the model criticism aspect of
Revised: With this in mind, we can consider how the model criticism aspect of

Original: RNPE could facilitate identification of the issue.
Revised: RNPE could facilitate identification of the issue.

Original: Fig.
Revised: Fig.

Original: 4a shows that most summary statistics are more likely to be well-specified than was believed a priori , with the clear exception of the Autocor summary statistic, which is inferred to be misspecified.
Revised: 4a shows that most summary statistics are more likely to be well-specified than was believed a priori , with the clear exception of the Autocor summary statistic, which is inferred to be misspecified.

Original: Fig.
Revised: Fig.

Original: 4b shows a density plot of the denoised data for the Autocor summary statistic (for a full pair plot, see Fig.
Revised: 4b shows a density plot of the denoised data for the Autocor summary statistic (for a full pair plot, see Fig.

Original: 12, in the appendix), showing that the observed autocorrelation is much lower than would be expected under the model.
Revised: 12, in the appendix), showing that the observed autocorrelation is much lower than would be expected under the model.

Original: This may prompt the practitioner to investigate the day-to-day changes in infections, in which case the discontinuities introduced by the reporting delays could be identified.
Revised: This may prompt the practitioner to investigate the day-to-day changes in infections, in which case the discontinuities introduced by the reporting delays could be identified.

Original: Fig.
Revised: Fig.

Original: 4c shows the posteriors for RNPE and NPE .
Revised: 4c shows the posteriors for RNPE and NPE .

Original: R NPE yields a posterior with far lower variance than NPE , demonstrating that it does not always achieve robustness simply by producing broader posterior estimates.
Revised: R NPE yields a posterior with far lower variance than NPE , demonstrating that it does not always achieve robustness simply by producing broader posterior estimates.

Original: [Figure]
Revised: [Figure]

Original: CS.
Revised: CS.

Original: In the CS model, the observed data was corrupted by removing cells in the core regions of tumours, mimicking necrosis (see Fig.
Revised: In the CS model, the observed data was corrupted by removing cells in the core regions of tumours, mimicking necrosis (see Fig.

Original: 9 in the appendix).
Revised: 9 in the appendix).

Original: This misspecification primarily impacts the N Cancer summary statistic (the number of cancer cells).
Revised: This misspecification primarily impacts the N Cancer summary statistic (the number of cancer cells).

Original: From Figs.
Revised: From Figs.

Original: 5a,5b, we can see that the results provide evidence that the N Cancer statistic is misspecified, with the observed data having fewer cancer cells than would be expected if it was generated from the simulator.
Revised: 5a,5b, we can see that the results provide evidence that the N Cancer statistic is misspecified, with the observed data having fewer cancer cells than would be expected if it was generated from the simulator.

Original: Once again, alongside domain knowledge, this information points the practitioner to the source of misspecification (necrosis), and hence the model could be adjusted appropriately.
Revised: Once again, alongside domain knowledge, this information points the practitioner to the source of misspecification (necrosis), and hence the model could be adjusted appropriately.

Original: For this example, NPE produced an overconfident posterior with little mass on the true parameters, whereas RNPE does not exhibit this issue (Fig. 5c).
Revised: For this example, NPE produced an overconfident posterior with little mass on the true parameters, whereas RNPE does not exhibit this issue (Fig. 5c).

Original: 5.3 Indeterminacy
Revised: 5.3 Indeterminacy

Original: In some cases, there can be indeterminacy when inferring misspecification, in which it is impossible to determine which summary statistic is misspecified, despite clear evidence of misspecification.
Revised: In some cases, there can be indeterminacy when inferring misspecification, in which it is impossible to determine which summary statistic is misspecified, despite clear evidence of misspecification.

Original: This occurs, for example, when a model can only be consistent with one observed summary statistic, e.g. x 1 , at the expense of being able to recreate another summary statistic, e.g. x 2 (or vice versa).
Revised: This occurs, for example, when a model can only be consistent with one observed summary statistic, e.g. x 1 , at the expense of being able to recreate another summary statistic, e.g. x 2 (or vice versa).

Original: R NPE allows these trade-offs to be identified, as the statistics will show a small frequency of being jointly well-specified, Pr ( z 1 = 0 , z
Revised: R NPE allows these trade-offs to be identified, as the statistics will show a small frequency of being jointly well-specified, Pr ( z 1 = 0 , z

Original: 2 = 0 | y ) , alongside reasonable probabilities of either statistic being well-specified, Pr ( z 1 = 1 , z 2 = 0 | y ) and Pr ( z 1 = 0 ,
Revised: 2 = 0 | y ) , alongside reasonable probabilities of either statistic being well-specified, Pr ( z 1 = 1 , z 2 = 0 | y ) and Pr ( z 1 = 0 ,

Original: z 2 = 1 | y ) .
Revised: z 2 = 1 | y ) .

Original: We show an example of this in
Revised: We show an example of this in

Original: Fig. 8 in the Appendix.
Revised: Fig. 8 in the Appendix.

Original: [Figure]
Revised: [Figure]

Original: p c d c
Revised: p c d c

Original:  Conclusions
Revised:  Conclusions

Original: We described RNPE , an approach that can be applied to any black-box simulator to simultaneously perform model criticism and robust inference .
Revised: We described RNPE , an approach that can be applied to any black-box simulator to simultaneously perform model criticism and robust inference .

Original: R NPE is philosophically similar to existing robust
Revised: R NPE is philosophically similar to existing robust

Original: SBI approaches for ABC and synthetic likelihood (Wilkinson, 2013; Frazier and Drovandi, 2021).
Revised: SBI approaches for ABC and synthetic likelihood (Wilkinson, 2013; Frazier and Drovandi, 2021).

Original: However, this work introduces three novel contributions: i) we apply the idea of using error models for robust inference to a neural SBI method, which has advantages in simulation efficiency, flexibility and scaling to high dimensional problems (Lueckmann et al., 2021), ii) we facilitate model criticism decoupled from inference, which avoids confounding of inference failures and model misspecification, and iii) we use a spike-and-slab error model to obtain explicit misspecification probability diagnostics for summary statistics, to facilitate model criticism .
Revised: However, this work introduces three novel contributions: i) we apply the idea of using error models for robust inference to a neural SBI method, which has advantages in simulation efficiency, flexibility and scaling to high dimensional problems (Lueckmann et al., 2021), ii) we facilitate model criticism decoupled from inference, which avoids confounding of inference failures and model misspecification, and iii) we use a spike-and-slab error model to obtain explicit misspecification probability diagnostics for summary statistics, to facilitate model criticism .

Original: Our work is not without limitations.
Revised: Our work is not without limitations.

Original: Firstly,
Revised: Firstly,

Original: RNPE is more computationally costly than NPE , requiring an additional MCMC step and fitting of q ( x ) .
Revised: RNPE is more computationally costly than NPE , requiring an additional MCMC step and fitting of q ( x ) .

Original: Secondly, we only evaluated performance of a single choice of error model on a limited number of tasks.
Revised: Secondly, we only evaluated performance of a single choice of error model on a limited number of tasks.

Original: It would be beneficial in the future to have a larger set of realistic misspecified tasks for assessing performance, similar to the work by Lueckmann et al.
Revised: It would be beneficial in the future to have a larger set of realistic misspecified tasks for assessing performance, similar to the work by Lueckmann et al.

Original: (2021) for the well-specified case.
Revised: (2021) for the well-specified case.

Original: Finally, we restricted our focus to NPE ; in future work it would be interesting to consider other SBI algorithms and to compare results with existing methods for robust inference.
Revised: Finally, we restricted our focus to NPE ; in future work it would be interesting to consider other SBI algorithms and to compare results with existing methods for robust inference.

Original: We demonstrated with four examples that RNPE consistently outperforms NPE for parameter inference under misspecification, tending to put more posterior mass on the true parameters whilst also exhibiting more conservative coverage properties.
Revised: We demonstrated with four examples that RNPE consistently outperforms NPE for parameter inference under misspecification, tending to put more posterior mass on the true parameters whilst also exhibiting more conservative coverage properties.

Original: The tasks were designed to mimic levels of misspecification that could be realistically encountered, and hence the poor performance of NPE calls into question its use in practice, particularly when the level of misspecification could be significant.
Revised: The tasks were designed to mimic levels of misspecification that could be realistically encountered, and hence the poor performance of NPE calls into question its use in practice, particularly when the level of misspecification could be significant.

Original: The substantial difference in performance highlights, as suggested by Box (1980), that we should not have blind faith that a model is sufficiently accurate, and therefore must devise methods for performing model criticism and robust inference .
Revised: The substantial difference in performance highlights, as suggested by Box (1980), that we should not have blind faith that a model is sufficiently accurate, and therefore must devise methods for performing model criticism and robust inference .

Original: With further research and wider adoption of these principles in SBI , we believe simulators will only become more powerful tools for scientific discovery.Appendix A Additional Results [Figure]
Revised: With further research and wider adoption of these principles in SBI , we believe simulators will only become more powerful tools for scientific discovery.

Original: In the classifier 2-sample test (

Original: Lopez-Paz and Oquab, 2016) a classifier is trained to distinguish between a true and inferred posterior distribution.

Original: In the case that the classifier struggles to distinguish between the samples (i.e. when the posterior approximation is good), the C2ST score approaches 0 .

Original: In contrast, when the posterior approximation is poor the C2ST score will be close to 1 .

Original: Here we take the ‘true’ posterior to be the analytical posterior obtained under the true data generating process.

Original: For calculation of the C2ST metric we used a multilayer perceptron, as implemented in the sbibm python package (Lueckmann et al., 2021).

Original: This metric requires a tractable posterior, so is only available for the Gaussian and Gaussian Linear tasks.

Original: We note that the error model used in RNPE and NNPE is misspecified, so we cannot realistically expect a C2ST score close to 0 .

Original: 5 , but the results are useful to highlight that even a misspecified error model can improve performance.

Original: These results are shown in Fig.

Original: Mean Square Error

Original: In addition to calculating the log-probability of θ ∗ and the posterior coverage properties (see Section 4.2 and Fig. 2), we also considered taking the posterior mean as a parameter point estimate and comparing this to the true parameters θ ∗ .

Original: Over the 1000 different true parameters and observation pairs, we compared the posterior means to the true parameters using the mean squared error

Original: [Equation]

Original: where ¯ θ ij is the posterior mean of the j -th parameter, resulting from either NPE or RNPE , and θ ∗ ij is the ground truth parameter sampled from the prior.

Original: These results are shown in Table

Original: [Table]

Original: [Table]

Original: [Table]

Original: Well-Specified Case

Original: An obvious question to ask is, what is the impact of incorrectly using an error model when the simulator is well-specified?

Original: These results are shown in Fig.

Original: In the well-specified case, NPE becomes well-calibrated (Fig. 7a), whereas RNPE produces slightly conservative posteriors.

Original: NPE performs marginally better in terms of the posterior mass placed on the true parameters (Fig. 7b).

Original: However, in the well-specified case, RNPE may indicate that the model is not likely to be misspecified, in which case the researcher could choose to collapse the error model by directly using q ( θ | x ) , with no additional cost.

Original: [Figure]

Original: B Tasks Gaussian

Original: Model Description.

Original: A simple Gaussian toy model used by Frazier et al. (2020b).

Original: The task has a single parameter µ , which is the mean of a Gaussian distribution.

Original: A simulation consists of drawing 100 i.i.d.

Original: samples from N ( µ, 1) , which are summarised to the sample mean and variance.

Original: Misspecification.

Original: To artificially introduce misspecification, samples were drawn from N ( µ, 2) for the observed data, rather than N ( µ, 1) , prior to summarising.

Original: Priors.

Original: µ ∼ N (0 , 25) .

Original: Gaussian Linear

Original: Model Description.

Original: A misspecified extension of a simple Gaussian toy model used by Lueckmann et al.

Original: The task involves inferring the mean vector θ of a 10-d Gaussian distribution, where a simulation consists of sampling this distribution i.e. x | θ ∼ N ( θ , 0 .

Original: 1 · I 10 ) .

Original: Misspecification.

Original: To artificially introduce misspecification, we add Gaussian additive noise such that y | x ∼ N ( x , 0 .

Original: [-1 · I 10 ) , or equivalently, we can view this as a misspecified variance in the simulator such that y | θ ∼ N ( θ , 0 . 
-]
Revised: [-1 · I 10 ) , or equivalently, we can view this as a misspecified variance in the simulator such that y | θ ∼ N ( θ , 0 . 
-]

Original: · I 10 ) .

Original: Priors.

Original: θ ∼ N (0 , 0 . 1 · I 10 ) .

Original: SIR

Original: Model Description.

Original: The SIR model is an idealised model of disease spread, where a series of differential equations describe the rates of transitions between three states: susceptible ( s ), infected ( i ) and recovered ( r ).

Original: The simplest (and deterministic) model is

Original: [Equation]

Original: where β is the infection rate, and γ the recovery rate.

Original: We use a stochastic extension of this model, with a time-varying infection rate ˜ β t (see Atkeson et al., 2020; Perla et al., 2022), reflecting for example policy changes or mutations in the virus.

Original: This process is parameterised using the basic reproduction number, R 0 t = ˜ β t γ

Original: [Equation]

Original: 11) where η is the mean reversion strength of R 0 t to βγ , σ is the volatility, and W t is Brownian motion.

Original: We set η = 0 .

Original: 05 and σ = 0 .

Original: 05 and focused on inferring β and γ .

Original: This model was implemented in Julia, using the DifferentialEquations package (Rackauckas and Nie, 2017).

Original: We scaled the resulting infection trajectories by 100,000 (representing the population size), and used summary statistics of the daily infection counts over 365 days, as described in Section 4.1.

Original: Misspecification.

Original: To artificially introduce misspecification, we reduce the infection counts on weekends by 5%, and these infections are recouped on the subsequent Monday, mimicking reporting delays over weekends.

Original: We (arbitrarily) take the first day to be Monday.

Original: An example of a simulations, compared to an observation with reporting delays is shown in Fig.

Original: Priors.

Original: We used β ∼ Uniform (0 , 0 . 5) and γ ∼ Uniform (0 , 0 . 5) , but rejected samples where γ > β as frequently the number of infections stayed near zero (as the recovery rate is greater than the infection rate).

Original: CS

Original: Model Description.

Original: The CS model simulates cancer and stromal cell development on a 2D plane.

Original: The total number of cells N c , number of unobserved parents N p , and the number of daughter cells for each parent N di are sampled

Original: [Equation]

Original: where λ c , λ p and λ d are the three Poisson rate parameters we attempt to infer.

Original: The cell positions { c i } N c

Original: i

Original: =

Original: and parent positions { p i } N p

Original: i

Original: =1 are generated uniformly on the 2D plane (i.e. as two homogeneous point processes).

Original: Let r i represent the euclidean distance from parent p i to its

Original: N di -th nearest cell.

Original: r i is the affected radius, such that cells are assigned to be cancerous if they fell on or within this distance of p i .

Original: We used summary statistics based on cell type counts and distances, as described in Section 4.1.

Original: In order to efficiently calculate the distances, we made use of the Numba just-in-time compiler for Python (Lam et al., 2015).

Original: The distance based metrics ( Mean Min Dist and Max Min Dist ) were approximated empirically through samplingstromal cells, as the full distance matrix was too expensive to compute.

Original: Misspecification.

Original: For each parent, we sampled w i ∼ Bernoulli (0 . 75) .

Original: If w i = 1 , cancer cells falling within 0 .

Original: 8 r i of the corresponding parent p i were removed.

Original: An example of this is shown in Fig.

Original: This mimics necrosis that often occurs in the centre regions of tumours (see e.g.,

Original: Jones-Todd et al., 2019).

Original: Priors.

Original: λ c ∼ Uniform (200 , 1500) ,

Original: λ p ∼ Uniform (3 , 20) , λ d ∼ Uniform (10 , 20) .

Original: C Hyperparameters

Original: Flows.

Original: For q ( x ) we used a block neural autoregressive flow (De Cao et al., 2020), with a single hidden layer of size 8 D , where x ∈ R D .

Original: For q ( θ | x ) we used a neural spline flow (Durkan et al.,

Original: , defining the transform on the interval [ − 5 , 5] , using 10 spline segments, and 5 coupling layers.

Original: For both flows we used a standard Gaussian base distribution.

Original: We used custom normalising flow implementations using JAX (Bradbury et al., 2018) and Equinox (Kidger and Garcia, 2021), which can be found at https://github.com/danielward27/flowjax .

Original: We found JAX flow implementations alongside NumPyro (Phan et al., 2019) for performing HMC to be several orders of magnitude faster than relying on Pytorch implementations and Pyro (Bingham et al., 2019).

Original: Training.

Original: For both flow approximations, we used a batch size of 256, using 10% of the data as a validation set.

Original: We trained for a maximum of 50 epochs, terminating training early if 5 consecutive epochs occurred with no improvement in the validation loss.

Original: For the block neural autoregressive flow q ( x ) , we used a learning rate of 1 × 10 − 2 , whereas for the neural spline flow q ( θ | x ) we used a learning rate of 5 × 10 − 4 .

Original: M CMC .

Original: We used 100,000 steps with 20,000 warm up steps.

Original: The trajectory length was set to 1, and the target acceptance probability to 0.95 to increase the robustness of the MCMC algorithm.

Original: All other hyperparameters were left to the defaults of the NumPyro Python package (Phan et al., 2019).

Original: D Computational Resources

Original: All experiments were performed on CPU, with a maximum of 8GB of RAM, using the computational facilities of the Advanced Computing Research Centre, University of Bristol - http: //www.bristol.ac.uk/acrc/ .

Original: E Additional Figures

Original: [Figure]

Original: Well-Specified

Original: Misspecified

Original: [Figure]

Original: , where we take p ( x ) to be a correlated Gaussian.

Original: (a) The denoised samples, compared to the observed data and simulations.

Original: (b) The frequencies with which the different pairwise combinations occurred in z ∼ ˆ p ( z | y ) , inferred during denoising.

Original: The results suggest a very low probability of both being well specified, despite reasonable probabilities of either x 1 or x 2 (exclusively) being well-specified, or both being misspecified.

Original: Despite the strong evidence for misspecification, the source of misspecification cannot be determined without further information.

Original: Observation

Original: Cancer cell

Original: Stromal cell 0.

Original: Simulation xx

Original: [Equation]

Original: x

Original: [Equation]

Original: x

Original: [Equation]

Original: xxxxxxxxxxxxx

Original: Simulations

Original: Denoised

Original: Observed

Original: Prior

Original: RNPE

Original: NPE

Original: True

Original: [Figure]

Original: [Figure]

Original: [Figure]