Original: Robust Neural Posterior Estimation and Statistical Model Criticism
Revised: Robust Neural Posterior Estimation and Statistical Model Criticism

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Computer simulations have proven a valuable tool for understanding complexphenomena across the sciences.
Revised: Computer simulations have proven a valuable tool for understanding complexphenomena across the sciences.

Original: However, the utility of simulators for modellingand forecasting purposes is often restricted by low data quality, as well as practicallimits to model fidelity.
Revised: However, the utility of simulators for modellingand forecasting purposes is often restricted by low data quality, as well as practicallimits to model fidelity.

Original: In order to circumvent these difficulties, we argue thatmodellers must treat simulators as idealistic representations of the true datagenerating process, and consequently should thoughtfully consider the risk ofmodel misspecification .
Revised: In order to circumvent these difficulties, we argue thatmodellers must treat simulators as idealistic representations of the true datagenerating process, and consequently should thoughtfully consider the risk ofmodel misspecification .

Original: In this work we revisit neural posterior estimation ( NPE ),a class of algorithms that enable black-box parameter inference in simulationmodels, and consider the implication of a simulation-to-reality gap.
Revised: In this work we revisit neural posterior estimation ( NPE ),a class of algorithms that enable black-box parameter inference in simulationmodels, and consider the implication of a simulation-to-reality gap.

Original: While recentworks have demonstrated reliable performance of these methods, the analyseshave been performed using synthetic data generated by the simulator model itself,and have therefore only addressed the well-specified case.
Revised: While recentworks have demonstrated reliable performance of these methods, the analyseshave been performed using synthetic data generated by the simulator model itself,and have therefore only addressed the well-specified case.

Original: In this paper, we findthat the presence of misspecification, in contrast, leads to unreliable inferencewhen NPE is used naïvely.
Revised: In this paper, we findthat the presence of misspecification, in contrast, leads to unreliable inferencewhen NPE is used naïvely.

Original: As a remedy we argue that principled scientific inquirywith simulators should incorporate a model criticism component, to facilitateinterpretable identification of misspecification and a robust inference component,to fit “wrong but useful” models.
Revised: As a remedy we argue that principled scientific inquirywith simulators should incorporate a model criticism component, to facilitateinterpretable identification of misspecification and a robust inference component,to fit “wrong but useful” models.

Original: We propose robust neural posterior estimation( RNPE ), an extension of NPE to simultaneously achieve both these aims, throughexplicitly modelling the discrepancies between simulations and the observeddata.
Revised: We propose robust neural posterior estimation( RNPE ), an extension of NPE to simultaneously achieve both these aims, throughexplicitly modelling the discrepancies between simulations and the observeddata.

Original: We assess the approach on a range of artificially misspecified examples, andfind RNPE performs well across the tasks, whereas naïvely using NPE leads tomisleading and erratic posteriors.
Revised: We assess the approach on a range of artificially misspecified examples, andfind RNPE performs well across the tasks, whereas naïvely using NPE leads tomisleading and erratic posteriors.

Original:  Introduction
Revised:  Introduction

Original: Stochastic simulators have become a ubiquitous modelling tool across the sciences and are regularlyapplied to some of the most complex and challenging problems of scientific interest, including climatechange (see e.g. Randall et al., 2007), particle physics (e.g. Brehmer et al., 2020) and the Covid-19pandemic (e.g. Ferguson et al., 2020).
Revised: Stochastic simulators have become a ubiquitous modelling tool across the sciences and are regularlyapplied to some of the most complex and challenging problems of scientific interest, including climatechange (see e.g. Randall et al., 2007), particle physics (e.g. Brehmer et al., 2020) and the Covid-19pandemic (e.g. Ferguson et al., 2020).

Original: Simulators implicitly define a likelihood function p ( x | θ ) ,where x is the simulator output and θ are the simulator parameters.
Revised: Simulators implicitly define a likelihood function p ( x | θ ) ,where x is the simulator output and θ are the simulator parameters.

Original: Although running the simulatorto sample from the model is straightforward, the inherent complexity of simulators often makesanalytic calculation of the likelihood intractable.
Revised: Although running the simulatorto sample from the model is straightforward, the inherent complexity of simulators often makesanalytic calculation of the likelihood intractable.

Original: As a result, classical inference techniques to find theparameter posterior p ( θ | x ) such as Markov chain Monte Carlo ( MCMC ) (Metropolis et al., 1953) areinfeasible.
Revised: As a result, classical inference techniques to find theparameter posterior p ( θ | x ) such as Markov chain Monte Carlo ( MCMC ) (Metropolis et al., 1953) areinfeasible.

Original: To overcome this issue, a large family of simulation-based inference ( SBI ) methods havebeen developed that allow parameter inference to be performed on arbitrary “black-box”
Revised: To overcome this issue, a large family of simulation-based inference ( SBI ) methods havebeen developed that allow parameter inference to be performed on arbitrary “black-box”

Original: simulators(see Cranmer et al., 2020).
Revised: simulators(see Cranmer et al., 2020).

Original: Recently, there has been much success applying neural network-basedconditional density estimators, such as normalising flows, to approximate the likelihood or posterior(Greenberg et al., 2019;
Revised: Recently, there has been much success applying neural network-basedconditional density estimators, such as normalising flows, to approximate the likelihood or posterior(Greenberg et al., 2019;

Original: Papamakarios and Murray, 2016; Papamakarios et al., 2019a; Lueckmann et al., 2021).
Revised: Papamakarios and Murray, 2016; Papamakarios et al., 2019a; Lueckmann et al., 2021).

Original: These approaches are broadly termed neural likelihood estimation ( NLE ) and neuralposterior estimation ( NPE ).
Revised: These approaches are broadly termed neural likelihood estimation ( NLE ) and neuralposterior estimation ( NPE ).

Original: Denoising
Revised: Denoising

Original: [Figure]
Revised: [Figure]

Original: Despite proving useful across the sciences, SBI methods are not without their limitations.
Revised: Despite proving useful across the sciences, SBI methods are not without their limitations.

Original: Firstly, evenin the ideal case of a well-specified simulator, these methods in practice lead to imperfect and oftenoverconfident posterior approximations (Hermans et al., 2021; Lueckmann et al., 2021).
Revised: Firstly, evenin the ideal case of a well-specified simulator, these methods in practice lead to imperfect and oftenoverconfident posterior approximations (Hermans et al., 2021; Lueckmann et al., 2021).

Original: Secondly,
Revised: Secondly,

Original: SBI methods generally rely on the assumption that the simulator is well-specified, an assumption thatis often violated in practice.
Revised: SBI methods generally rely on the assumption that the simulator is well-specified, an assumption thatis often violated in practice.

Original: It is this second issue that forms the focus of this work.
Revised: It is this second issue that forms the focus of this work.

Original: To define misspecification, it is necessary to make a clear distinction between the true data generatingprocess and the simulator: we will write y o for observed data, following some unknown data generating process with distribution denoted p ∗ , and we will write x ∼ p ( x | θ ) to denote samples fromthe simulator.
Revised: To define misspecification, it is necessary to make a clear distinction between the true data generatingprocess and the simulator: we will write y o for observed data, following some unknown data generating process with distribution denoted p ∗ , and we will write x ∼ p ( x | θ ) to denote samples fromthe simulator.

Original: In many applications of SBI , summary statistics are used to reduce the dimensionalityof the problem.
Revised: In many applications of SBI , summary statistics are used to reduce the dimensionalityof the problem.

Original: If this is the case, we use x and y o to denote the simulated and observed summarystatistics, rather than the raw data, and treat the raw data as an unobserved latent of the model.
Revised: If this is the case, we use x and y o to denote the simulated and observed summarystatistics, rather than the raw data, and treat the raw data as an unobserved latent of the model.

Original: Thesimulator is said to be misspecified if the true data generating process does not fall within the familyof distributions defined by the simulator, i.e. p ∗ / ∈ { p ( x | θ ); θ ∈ Θ } .
Revised: Thesimulator is said to be misspecified if the true data generating process does not fall within the familyof distributions defined by the simulator, i.e. p ∗ / ∈ { p ( x | θ ); θ ∈ Θ } .

Original: Simulators generally formidealised and simplistic representations of a more complex data generating process, and hence – likeall statistical models – will be “wrong” to some extent.
Revised: Simulators generally formidealised and simplistic representations of a more complex data generating process, and hence – likeall statistical models – will be “wrong” to some extent.

Original: This misspecification often results in adiscrepancy between simulated and observed data, sometimes termed a “simulation-to-reality gap”.
Revised: This misspecification often results in adiscrepancy between simulated and observed data, sometimes termed a “simulation-to-reality gap”.

Original: Despite this, the general approach in SBI is to learn a posterior approximation q ( θ | x ) (either directly,or via the likelihood), trained using simulated data, ( x , θ ) ∼ p ( x | θ ) p ( θ ) , and to query it with theobserved data, q ( θ | x = y o ) .
Revised: Despite this, the general approach in SBI is to learn a posterior approximation q ( θ | x ) (either directly,or via the likelihood), trained using simulated data, ( x , θ ) ∼ p ( x | θ ) p ( θ ) , and to query it with theobserved data, q ( θ | x = y o ) .

Original: This approach is only theoretically justified under the assumptionthat the simulator is well-specified; relying on this approach for misspecified simulators leads to twoissues:
Revised: This approach is only theoretically justified under the assumptionthat the simulator is well-specified; relying on this approach for misspecified simulators leads to twoissues:

Original: (i)
Revised: (i)

Original: As the posterior approximation q ( θ | x ) is learned using samples from the simulator, this
Revised: As the posterior approximation q ( θ | x ) is learned using samples from the simulator, this

Original: implies that we can only expect it to form a reasonable approximation of p ( θ | x ) for regions
Revised: implies that we can only expect it to form a reasonable approximation of p ( θ | x ) for regions

Original: well covered by simulations.
Revised: well covered by simulations.

Original: For misspecified simulators, the observed data y o may be very
Revised: For misspecified simulators, the observed data y o may be very

Original: unlikely in p ( x ) = (cid:82) p ( x | θ ) p ( θ ) d θ , in which case the approximation q ( θ | x
Revised: unlikely in p ( x ) = (cid:82) p ( x | θ ) p ( θ ) d θ , in which case the approximation q ( θ | x

Original: = y o ) will
Revised: = y o ) will

Original: likely be poor.
Revised: likely be poor.

Original: In some cases, y o may even fall outside the support of p ( x ) , which results in
Revised: In some cases, y o may even fall outside the support of p ( x ) , which results in

Original: NPE attempting to extrapolate to estimate the undefined posterior.
Revised: NPE attempting to extrapolate to estimate the undefined posterior.

Original: (ii) It has been demonstrated more generally that Bayesian inference is frequently problematic
Revised: (ii) It has been demonstrated more generally that Bayesian inference is frequently problematic

Original: under misspecification.
Revised: under misspecification.

Original: Even for simple models, misspecification can lead to posterior con-
Revised: Even for simple models, misspecification can lead to posterior con-

Original: centration around “bad” models (Grünwald and Langford, 2007; Grünwald and Van Ommen,
Revised: centration around “bad” models (Grünwald and Langford, 2007; Grünwald and Van Ommen,

Original: 2017) and poorly calibrated credible regions (Kleijn and van der Vaart, 2012; Syring and
Revised: 2017) and poorly calibrated credible regions (Kleijn and van der Vaart, 2012; Syring and

Original: Martin, 2019).
Revised: Martin, 2019).

Original: These issues will persist even in the unlikely case that q ( θ | x ) perfectly
Revised: These issues will persist even in the unlikely case that q ( θ | x ) perfectly

Original: approximates p ( θ | x ) .
Revised: approximates p ( θ | x ) .

Original: We view the ubiquity of model misspecification (to a generally unknown degree), in addition to thelack of principled methods for addressing it, as a major obstacle to further adoption of SBI .
Revised: We view the ubiquity of model misspecification (to a generally unknown degree), in addition to thelack of principled methods for addressing it, as a major obstacle to further adoption of SBI .

Original: Our contribution.
Revised: Our contribution.

Original: We propose robust neural posterior estimation ( RNPE ), an SBI framework thatincorporates an error model, p ( y | x ) , to explicitly express our belief that a discrepancy mightexist between the data generating process and the simulator.
Revised: We propose robust neural posterior estimation ( RNPE ), an SBI framework thatincorporates an error model, p ( y | x ) , to explicitly express our belief that a discrepancy mightexist between the data generating process and the simulator.

Original: Following ideas from Box (1980),
Revised: Following ideas from Box (1980),

Original: RNPE facilitates principled scientific modelling, allowing iteration between two complementarycomponents: model criticism 1 and robust inference :
Revised: RNPE facilitates principled scientific modelling, allowing iteration between two complementarycomponents: model criticism 1 and robust inference :

Original: Model Criticism.
Revised: Model Criticism.

Original: Model criticism has long been seen as a key component in statistical modelling; itis only through understanding the limitations of a model that principled model development canoccur (Box, 1980; Gelman and Shalizi, 2013).
Revised: Model criticism has long been seen as a key component in statistical modelling; itis only through understanding the limitations of a model that principled model development canoccur (Box, 1980; Gelman and Shalizi, 2013).

Original: The error model provides a means to comparethe observed data to simulations, facilitating automated and interpretable identification ofmisspecification.
Revised: The error model provides a means to comparethe observed data to simulations, facilitating automated and interpretable identification ofmisspecification.

Original: Robust Inference.
Revised: Robust Inference.

Original: Even after several rounds of model development and model criticism, the “best”available model will still frequently be misspecified to an appreciable degree.
Revised: Even after several rounds of model development and model criticism, the “best”available model will still frequently be misspecified to an appreciable degree.

Original: However, it is oftenstill desirable to use “wrong but useful” models for parameter inference and predictions, andhence the inference process should be robust to misspecification.
Revised: However, it is oftenstill desirable to use “wrong but useful” models for parameter inference and predictions, andhence the inference process should be robust to misspecification.

Original: R NPE partitions inference intotwo steps: i) probabilistically “denoising” the observation, thus producing samples consistentwith both the error model and simulator; ii) performing NPE , using the property of amortisation 2to ensemble posterior estimates over the denoised data.
Revised: R NPE partitions inference intotwo steps: i) probabilistically “denoising” the observation, thus producing samples consistentwith both the error model and simulator; ii) performing NPE , using the property of amortisation 2to ensemble posterior estimates over the denoised data.

Original: In this manner, we incorporate thepossibility of model misspecification into the Bayesian update, whilst simultaneously bypassingthe need to rely on the ability of NPE to generalise well to misspecified observations.
Revised: In this manner, we incorporate thepossibility of model misspecification into the Bayesian update, whilst simultaneously bypassingthe need to rely on the ability of NPE to generalise well to misspecified observations.

Original: An overview of the RNPE framework can be seen in Fig.
Revised: An overview of the RNPE framework can be seen in Fig.

Original: We examine the practical value of thisapproach for model criticism and robust inference on examples in which we artificially introducerealistic levels of misspecification.
Revised: We examine the practical value of thisapproach for model criticism and robust inference on examples in which we artificially introducerealistic levels of misspecification.

Original:  Method2.1 Error Model
Revised:  Method2.1 Error Model

Original: As outlined above, we argue that if we are interested in being able to draw trustworthy scientificconclusions, we should ensure the modelling process accommodates both model criticism and robustinference .
Revised: As outlined above, we argue that if we are interested in being able to draw trustworthy scientificconclusions, we should ensure the modelling process accommodates both model criticism and robustinference .

Original: To this end, we explicitly account for possible discrepancies between the simulated andobserved data by incorporating an “error model” p ( y | x ) , such that our new assumed generativemodel is
Revised: To this end, we explicitly account for possible discrepancies between the simulated andobserved data by incorporating an “error model” p ( y | x ) , such that our new assumed generativemodel is

Original: [Equation]
Revised: [Equation]

Original: where we assume that the error distribution is independent of the simulator parameters θ .
Revised: where we assume that the error distribution is independent of the simulator parameters θ .

Original: Using theequivalent factorisation p ( θ , x , y )
Revised: Using theequivalent factorisation p ( θ , x , y )

Original: = p ( y ) p ( x | y ) p ( θ | x ) , and marginalising over x yields
Revised: = p ( y ) p ( x | y ) p ( θ | x ) , and marginalising over x yields

Original: [Equation]
Revised: [Equation]

Original: Dividing by the marginal likelihood p ( y ) , we obtain an expression for the parameter posterior
Revised: Dividing by the marginal likelihood p ( y ) , we obtain an expression for the parameter posterior

Original: [Equation]
Revised: [Equation]

Original: Equation (3) involves two unknown distributions, p ( x | y ) and p ( θ | x ) , which we require forinference.
Revised: Equation (3) involves two unknown distributions, p ( x | y ) and p ( θ | x ) , which we require forinference.

Original: The distribution p ( θ | x ) can be approximated by training a normalising flow q ( θ | x ) onsimulated data ( x , θ ) ∼ p ( x | θ ) p ( θ ) , as is done for NPE .
Revised: The distribution p ( θ | x ) can be approximated by training a normalising flow q ( θ | x ) onsimulated data ( x , θ ) ∼ p ( x | θ ) p ( θ ) , as is done for NPE .

Original: To approximately sample from p ( x | y ) ,we first train a normalising flow, q ( x ) , fitted to samples from prior predictive distribution of thesimulator x ∼ p ( x ) .
Revised: To approximately sample from p ( x | y ) ,we first train a normalising flow, q ( x ) , fitted to samples from prior predictive distribution of thesimulator x ∼ p ( x ) .

Original: Given an error model p ( y | x ) specified a priori , we can then approximatelysample from p ( x | y ) using MCMC , as ˆ p ( x | y )
Revised: Given an error model p ( y | x ) specified a priori , we can then approximatelysample from p ( x | y ) using MCMC , as ˆ p ( x | y )

Original: ∝ p ( y | x ) q ( x ) .
Revised: ∝ p ( y | x ) q ( x ) .

Original: With these approximations, we cansample the robust posterior ˆ p ( θ | y ) by first sampling ˜ x ∼ ˆ p ( x | y ) , then sampling ˜ θ ∼ q ( θ | ˜ x ) , discarding the latent variable ˜ x , where the tilde is used to distinguish from prior samples andsimulations.
Revised: With these approximations, we cansample the robust posterior ˆ p ( θ | y ) by first sampling ˜ x ∼ ˆ p ( x | y ) , then sampling ˜ θ ∼ q ( θ | ˜ x ) , discarding the latent variable ˜ x , where the tilde is used to distinguish from prior samples andsimulations.

Original: A Monte Carlo approximation of the expectation in Equation (3) can be used to estimatethe posterior density
Revised: A Monte Carlo approximation of the expectation in Equation (3) can be used to estimatethe posterior density

Original: ˆ p ( θ | y ) , i.e.
Revised: ˆ p ( θ | y ) , i.e.

Original: [Equation]
Revised: [Equation]

Original: We will refer to ˆ p ( x | y o ) as the denoised data distribution, as it intuitively finds a compromisebetween being close to the observed data y o , as measured by the error model p ( y | x ) , but consistentwith the simulator, as measured by q ( x ) .
Revised: We will refer to ˆ p ( x | y o ) as the denoised data distribution, as it intuitively finds a compromisebetween being close to the observed data y o , as measured by the error model p ( y | x ) , but consistentwith the simulator, as measured by q ( x ) .

Original: This formulation becomes natural when considering thegenerative model in (1).
Revised: This formulation becomes natural when considering thegenerative model in (1).

Original: We are (approximately) reversing the new assumed data generating processwhilst propagating uncertainty throughout, first removing the discrepancy between simulations andthe observed data by denoising, and then finding the associated simulator parameters.
Revised: We are (approximately) reversing the new assumed data generating processwhilst propagating uncertainty throughout, first removing the discrepancy between simulations andthe observed data by denoising, and then finding the associated simulator parameters.

Original: Under thisframework, the standard SBI approach can be retrieved as the special case in which p ( y | x ) =p ( x | y )
Revised: Under thisframework, the standard SBI approach can be retrieved as the special case in which p ( y | x ) =p ( x | y )

Original: =
Revised: =

Original: δ ( x − y ) , where δ is the Dirac delta distribution, meaning that we revert to assuming nodiscrepancy between the simulator and data generating process.
Revised: δ ( x − y ) , where δ is the Dirac delta distribution, meaning that we revert to assuming nodiscrepancy between the simulator and data generating process.

Original: Pseudo-code for the overall approachis given in Algorithm 1.
Revised: Pseudo-code for the overall approachis given in Algorithm 1.

Original: end return { x m ) }
Revised: end return { x m ) }

Original: Mm
Revised: Mm

Original: =1 , samples approximately from p ( θ , x | y o )
Revised: =1 , samples approximately from p ( θ , x | y o )

Original: The idealised nature of simulation models implies that frequently there will be some characteristicsof the observed data that are well captured by simulations, and some aspects in which there existsa discrepancy.
Revised: The idealised nature of simulation models implies that frequently there will be some characteristicsof the observed data that are well captured by simulations, and some aspects in which there existsa discrepancy.

Original: Using a set of summary statistics for inference, rather than the raw data naturallycaptures an interpretable, low dimensional and diverse set of characteristics of the simulated andobserved data.
Revised: Using a set of summary statistics for inference, rather than the raw data naturallycaptures an interpretable, low dimensional and diverse set of characteristics of the simulated andobserved data.

Original: Because of this, we find that in general, reducing the raw data to summary statisticsis advantageous for model criticism , as the practitioner can choose summary statistics which alignwith their belief about important model characteristics, and easily interpret any discrepancies (see
Revised: Because of this, we find that in general, reducing the raw data to summary statisticsis advantageous for model criticism , as the practitioner can choose summary statistics which alignwith their belief about important model characteristics, and easily interpret any discrepancies (see

Original: Section 4.4).
Revised: Section 4.4).

Original: The requirement for domain specific knowledge to develop summary statistics has beenalleviated by recent work (e.g. Fearnhead and Prangle, 2012;
Revised: The requirement for domain specific knowledge to develop summary statistics has beenalleviated by recent work (e.g. Fearnhead and Prangle, 2012;

Original: Chan et al., 2018; Chen et al.
Revised: Chan et al., 2018; Chen et al.

Original: , 2020),which seek to automatically embed the raw data to summary statistics; however, these embeddingsoften lack a tangible interpretation.
Revised: , 2020),which seek to automatically embed the raw data to summary statistics; however, these embeddingsoften lack a tangible interpretation.

Original: Alternatively, both hand-crafted and automatic summary statisticscould be used in tandem.
Revised: Alternatively, both hand-crafted and automatic summary statisticscould be used in tandem.

Original: Hereafter, however, we perform model criticism and inference using handcrafted summary statistics, such that x and y o refer to simulated and observed summary statistics,not the raw data.
Revised: Hereafter, however, we perform model criticism and inference using handcrafted summary statistics, such that x and y o refer to simulated and observed summary statistics,not the raw data.

Original: 2.2 Spike and Slab
Revised: 2.2 Spike and Slab

Original: An error model should describe our prior belief over the discrepancy between the data generatingprocess and simulations.
Revised: An error model should describe our prior belief over the discrepancy between the data generatingprocess and simulations.

Original: Additionally, to facilitate model criticism, it should ideally allow easyassessment of which summary statistics are approximately well-specified and which are misspecified.
Revised: Additionally, to facilitate model criticism, it should ideally allow easyassessment of which summary statistics are approximately well-specified and which are misspecified.

Original: Inspired by sparsity-inducing priors used in Bayesian regression literature (George and McCulloch,1993), we use a “spike and slab” error model on each summary statistic
Revised: Inspired by sparsity-inducing priors used in Bayesian regression literature (George and McCulloch,1993), we use a “spike and slab” error model on each summary statistic

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: , D (8) where x ∈ R D .
Revised: , D (8) where x ∈ R D .

Original: As the summary statistics have varying scales, we standardise them to have meanzero and variance one.
Revised: As the summary statistics have varying scales, we standardise them to have meanzero and variance one.

Original: The Bernoulli variables indicate whether the j -th summary statistic isapproximately well-specified; we use the probability of ρ = 0 .
Revised: The Bernoulli variables indicate whether the j -th summary statistic isapproximately well-specified; we use the probability of ρ = 0 .

Original: 5 to express the belief that it is equallylikely a summary statistic will be misspecified or well-specified a priori .
Revised: 5 to express the belief that it is equallylikely a summary statistic will be misspecified or well-specified a priori .

Original: Alternatively, a hyperpriorcould be considered, but here we do not pursue this idea.
Revised: Alternatively, a hyperpriorcould be considered, but here we do not pursue this idea.

Original: When a summary statistic is well-specified(i.e.
Revised: When a summary statistic is well-specified(i.e.

Original: z j = 0 ), we take the error distribution to be a tight Gaussian (a spike) centered on x j , choosingσ = 0 .
Revised: z j = 0 ), we take the error distribution to be a tight Gaussian (a spike) centred on x j , choosingσ = 0 .

Original: 01 to enforce consistency between the simulator and the observed data for the j -th statistic.
Revised: 01 to enforce consistency between the simulator and the observed data for the j -th statistic.

Original: Incontrast, in the misspecified case (i.e., z j = 1 ), we take the error model to be the much wider andheavy tailed Cauchy distribution (a slab).
Revised: Incontrast, in the misspecified case (i.e., z j = 1 ), we take the error model to be the much wider andheavy tailed Cauchy distribution (a slab).

Original: Generally, we might expect misspecification to be relativelysubtle, but some model inadequacies can lead to catastrophically misspecified summary statistics.
Revised: Generally, we might expect misspecification to be relativelysubtle, but some model inadequacies can lead to catastrophically misspecified summary statistics.

Original: We chose the Cauchy scale τ = 0 .
Revised: We chose the Cauchy scale τ = 0 .

Original: 25 , to reflect this, as it places half the mass within ± 0 .
Revised: 25 , to reflect this, as it places half the mass within ± 0 .

Original: 25 standarddeviations of x j , but the long tails accommodate summary statistics that are very misspecified.
Revised: 25 standarddeviations of x j , but the long tails accommodate summary statistics that are very misspecified.

Original: Bymarginalising over z , the spike and slab error model can also be written as
Revised: Bymarginalising over z , the spike and slab error model can also be written as

Original: [Equation]
Revised: [Equation]

Original: ) and as such can be viewed as an equally weighted mixture of the Gaussian spike, p ( y j | x j , z j = 0) ,and the Cauchy slab, p ( y j | x j , z j = 1) , for each summary statistic.
Revised: ) and as such can be viewed as an equally weighted mixture of the Gaussian spike, p ( y j | x j , z j = 0) ,and the Cauchy slab, p ( y j | x j , z j = 1) , for each summary statistic.

Original: We note that error modelhyperparameter tuning approaches could be considered, e.g. a reasonable heuristic would be tochoose an error model that is broad enough that the denoised data ˜ x tend not to be outliers in q ( x ) ,compared to simulations 3 .
Revised: We note that error modelhyperparameter tuning approaches could be considered, e.g. a reasonable heuristic would be tochoose an error model that is broad enough that the denoised data ˜ x tend not to be outliers in q ( x ) ,compared to simulations 3 .

Original: However, we chose to keep the hyperparameters consistent across tasks, toavoid the risk of overfitting and to demonstrate that neither strong prior knowledge on the error model,nor careful hyperparameter tuning is necessary to yield substantial improvements in performance.
Revised: However, we chose to keep the hyperparameters consistent across tasks, toavoid the risk of overfitting and to demonstrate that neither strong prior knowledge on the error model,nor careful hyperparameter tuning is necessary to yield substantial improvements in performance.

Original: A key advantage of the spike and slab error model is given by the latent variable z .
Revised: A key advantage of the spike and slab error model is given by the latent variable z .

Original: Similar to posteriorinclusion probabilities in Bayesian regression, the posterior frequency of z j = 1 , p ( z j = 1 | y ) , canbe used as an indicator of the posterior misspecification probability for the j -th summary statistic.
Revised: Similar to posteriorinclusion probabilities in Bayesian regression, the posterior frequency of z j = 1 , p ( z j = 1 | y ) , canbe used as an indicator of the posterior misspecification probability for the j -th summary statistic.

Original: By comparing to the prior misspecification probability ρ , we can say that if p ( z j = 1 | y ) > ρ , themodel provides evidence of misspecification for the j -th summary statistic, and if p ( z j = 1 | y ) <
Revised: By comparing to the prior misspecification probability ρ , we can say that if p ( z j = 1 | y ) > ρ , themodel provides evidence of misspecification for the j -th summary statistic, and if p ( z j = 1 | y ) <

Original: ρ ,it provides evidence it is well-specified (Talbott, 2016).
Revised: ρ ,it provides evidence it is well-specified (Talbott, 2016).

Original: For the purpose of generality, the latentvariable z was not included when describing RNPE thus far.
Revised: For the purpose of generality, the latentvariable z was not included when describing RNPE thus far.

Original: However, we can jointly infer theposterior ˆ p ( x , z | y ) in step 5 of algorithm 1, by using an MCMC algorithm that supports samplingboth continuous and discrete variables.
Revised: However, we can jointly infer theposterior ˆ p ( x , z | y ) in step 5 of algorithm 1, by using an MCMC algorithm that supports samplingboth continuous and discrete variables.

Original: For this purpose, we used mixed Hamiltonian Monte Carlo( HMC ) (Zhou, 2020, 2022) from the NumPyro python package (Phan et al., 2019).
Revised: For this purpose, we used mixed Hamiltonian Monte Carlo( HMC ) (Zhou, 2020, 2022) from the NumPyro python package (Phan et al., 2019).

Original:  Related Work
Revised:  Related Work

Original: 3.1 Model Criticism in SBI
Revised: 3.1 Model Criticism in SBI

Original: Posterior predictive checks have been widely used in SBI , to compare the predictive samples to theobserved data (e.g. Durkan et al., 2020; Greenberg et al., 2019; Papamakarios et al., 2019b).
Revised: Posterior predictive checks have been widely used in SBI , to compare the predictive samples to theobserved data (e.g. Durkan et al., 2020; Greenberg et al., 2019; Papamakarios et al., 2019b).

Original: Althoughthis is a form of model criticism, we note a key limitation of this approach is that if discrepancy isdiscovered, it may not be clear whether this is due to the failure of the inference procedure, or due tosimulator misspecification.
Revised: Althoughthis is a form of model criticism, we note a key limitation of this approach is that if discrepancy isdiscovered, it may not be clear whether this is due to the failure of the inference procedure, or due tosimulator misspecification.

Original: In general, it may be possible to identify the presence of misspecificationusing anomaly/novelty detection.
Revised: In general, it may be possible to identify the presence of misspecificationusing anomaly/novelty detection.

Original: One such approach was suggested by Schmitt et al.
Revised: One such approach was suggested by Schmitt et al.

Original: (2021) in thecontext of SBI .
Revised: (2021) in thecontext of SBI .

Original: However, this approach does not facilitate interpreting misspecification, and it is notclear how to extend it to perform robust inference.
Revised: However, this approach does not facilitate interpreting misspecification, and it is notclear how to extend it to perform robust inference.

Original: There are a few related methods that criticise the a posterior estimate by assessing its calibrationusing simulated data (e.g. Hermans et al., 2021; Prangle et al., 2014; Talts et al., 2018); however,these results are only applicable under the assumption the model is well-specified.
Revised: There are a few related methods that criticise the a posterior estimate by assessing its calibrationusing simulated data (e.g. Hermans et al., 2021; Prangle et al., 2014; Talts et al., 2018); however,these results are only applicable under the assumption the model is well-specified.

Original: 3.2 Robust Inference in SBI
Revised: 3.2 Robust Inference in SBI

Original: Approximate Bayesian computation ( ABC ) is a family of SBI methods, characterised by their useof rejection sampling alongside a kernel function to compare the simulated and observed data
Revised: Approximate Bayesian computation ( ABC ) is a family of SBI methods, characterised by their useof rejection sampling alongside a kernel function to compare the simulated and observed data

Original: (Tavaré et al., 1997; Pritchard et al., 1999; Beaumont et al., 2002).
Revised: (Tavaré et al., 1997; Pritchard et al., 1999; Beaumont et al., 2002).

Original: The simplest form of ABC usessimulations with θ ∼ p ( θ ) , and accepts parameters with probability proportional to the similarity ofthe corresponding simulations to the observed data.
Revised: The simplest form of ABC usessimulations with θ ∼ p ( θ ) , and accepts parameters with probability proportional to the similarity ofthe corresponding simulations to the observed data.

Original: The use of a kernel is often seen as a necessaryevil, as enforcing an exact match between the observed and simulated data generally results in allsamples being rejected.
Revised: The use of a kernel is often seen as a necessaryevil, as enforcing an exact match between the observed and simulated data generally results in allsamples being rejected.

Original: However, it has been noted that the kernel function can often be interpretedas specifying an error model distribution p ( y | x ) (Wilkinson, 2013).
Revised: However, it has been noted that the kernel function can often be interpretedas specifying an error model distribution p ( y | x ) (Wilkinson, 2013).

Original: This idea has been expandedusing generalised Bayesian inference, in which the kernel is replaced with a loss function (Schmonet al., 2020), which can be chosen to use a robust measure of discrepancy such as the maximum meandiscrepancy ( MMD )
Revised: This idea has been expandedusing generalised Bayesian inference, in which the kernel is replaced with a loss function (Schmonet al., 2020), which can be chosen to use a robust measure of discrepancy such as the maximum meandiscrepancy ( MMD )

Original: (Chérief-Abdellatif and Alquier, 2020;
Revised: (Chérief-Abdellatif and Alquier, 2020;

Original: Frazier et al., 2020a; Park et al., 2015).
Revised: Frazier et al., 2020a; Park et al., 2015).

Original: Similar ideas have also recently been applied by Dellaporta et al.
Revised: Similar ideas have also recently been applied by Dellaporta et al.

Original: (2022), who adapt non-parametriclearning (see Lyddon et al., 2018) to SBI , to approximately sample a posterior which minimises
Revised: (2022), who adapt non-parametriclearning (see Lyddon et al., 2018) to SBI , to approximately sample a posterior which minimises

Original: the
Revised: the

Original: MMD between the model and data generating process.
Revised: MMD between the model and data generating process.

Original: However, sampling this approximate posteriorrequires optimizing a non-convex MMD objective.
Revised: However, sampling this approximate posteriorrequires optimizing a non-convex MMD objective.

Original: Synthetic likelihood is an SBI method that uses a Gaussian approximation of the likelihood (Wood,2010).
Revised: Synthetic likelihood is an SBI method that uses a Gaussian approximation of the likelihood (Wood,2010).

Original: Frazier and Drovandi (2021) investigated a robust synthetic likelihood algorithm, in which thevariance components of the Gaussian likelihood are expanded based on additional free parameters,inferred in a nested MCMC sampling scheme.
Revised: Frazier and Drovandi (2021) investigated a robust synthetic likelihood algorithm, in which thevariance components of the Gaussian likelihood are expanded based on additional free parameters,inferred in a nested MCMC sampling scheme.

Original: In the context of the paper, they interpret this approachas “likelihood adjustment”; however, we note this could equivalently be reframed as the addition ofan independent Gaussian error model, with a prior on the variances 4 .
Revised: In the context of the paper, they interpret this approachas “likelihood adjustment”; however, we note this could equivalently be reframed as the addition ofan independent Gaussian error model, with a prior on the variances 4 .

Original: In contrast to more traditional SBI methods, few solutions have been proposed for performing robustinference with neural SBI methods.
Revised: In contrast to more traditional SBI methods, few solutions have been proposed for performing robustinference with neural SBI methods.

Original: Perhaps the most common approach is to augment the modelwith additional nuisance parameters (e.g. additive noise), to expand the range of values the simulatorcan reasonably produce (Cranmer et al., 2020).
Revised: Perhaps the most common approach is to augment the modelwith additional nuisance parameters (e.g. additive noise), to expand the range of values the simulatorcan reasonably produce (Cranmer et al., 2020).

Original: This again can generally be interpreted as a naïvemethod of incorporating an error model, and does not facilitate model criticism .
Revised: This again can generally be interpreted as a naïvemethod of incorporating an error model, and does not facilitate model criticism .

Original:  Experiments
Revised:  Experiments

Original: We benchmark the performance of RNPE for parameter inference for three misspecified tasks, comparing results to NPE .
Revised: We benchmark the performance of RNPE for parameter inference for three misspecified tasks, comparing results to NPE .

Original: To reliably assess performance, we repeat the inference procedure with 1000different observations and ground truth parameter pairs for each misspecified task, and assess theperformance based on two metrics: i) the log probability of the true parameter vector, θ ∗ , and ii)the posterior coverage properties (see Section 4.3).
Revised: To reliably assess performance, we repeat the inference procedure with 1000different observations and ground truth parameter pairs for each misspecified task, and assess theperformance based on two metrics: i) the log probability of the true parameter vector, θ ∗ , and ii)the posterior coverage properties (see Section 4.3).

Original: We further show examples of RNPE on a singleobservation for each task.
Revised: We further show examples of RNPE on a singleobservation for each task.

Original: Looking at specific examples demonstrates how RNPE can be used in apractical setting to perform model criticism and robust posterior inference.
Revised: Looking at specific examples demonstrates how RNPE can be used in apractical setting to perform model criticism and robust posterior inference.

Original: The code required toreproduce all the results from this manuscript is available at REDACTED_FOR_ANONYMITY .
Revised: The code required toreproduce all the results from this manuscript is available at REDACTED_FOR_ANONYMITY .

Original: 4.1 Setup
Revised: 4.1 Setup

Original: For all experiments, we used N = 50 , 000 simulations, with M = 100 , 000 MCMC samples following20,000 warm up steps.
Revised: For all experiments, we used N = 50 , 000 simulations, with M = 100 , 000 MCMC samples following20,000 warm up steps.

Original: The MCMC chains were initialised using a random simulation, and z j =1 for j = 1 , . . .
Revised: The MCMC chains were initialised using a random simulation, and z j =1 for j = 1 , . . .

Original: , D .
Revised: , D .

Original: To build the approximation q ( x ) , we used block neural autoregressive flows(De Cao et al., 2020), as we do not require the ability to sample directly from q ( x ) .
Revised: To build the approximation q ( x ) , we used block neural autoregressive flows(De Cao et al., 2020), as we do not require the ability to sample directly from q ( x ) .

Original: For theapproximation of q ( θ | x ) , we utilised neural spline flows (Durkan et al., 2019).
Revised: For theapproximation of q ( θ | x ) , we utilised neural spline flows (Durkan et al., 2019).

Original: For all tasksthe hyperparameters were kept consistent; information on hyperparameter choices can be found in
Revised: For all tasksthe hyperparameters were kept consistent; information on hyperparameter choices can be found in

Original: Appendix A.3.
Revised: Appendix A.3.

Original: 4.2 Tasks
Revised: 4.2 Tasks

Original: Detailed descriptions of each task can be found in Appendix A.2.
Revised: Detailed descriptions of each task can be found in Appendix A.2.

Original: Gaussian.
Revised: Gaussian.

Original: A simple Gaussian example from Frazier et al.
Revised: A simple Gaussian example from Frazier et al.

Original: (2020b).
Revised: (2020b).

Original: The task involves predicting asingle parameter µ , using the sample mean and variance of 100 i.i.d.
Revised: The task involves predicting asingle parameter µ , using the sample mean and variance of 100 i.i.d.

Original: samples drawn from N ( µ, 1) ,where 1 is the variance of the Gaussian distribution.
Revised: samples drawn from N ( µ, 1) ,where 1 is the variance of the Gaussian distribution.

Original: To artificially produce misspecified observations,the 100 i.i.d.
Revised: To artificially produce misspecified observations,the 100 i.i.d.

Original: are instead drawn from N ( µ, 2) .
Revised: are instead drawn from N ( µ, 2) .

Original: Despite its simplicity, this task usefully demonstratesthat issues with misspecification can arise even in the simplest of models.
Revised: Despite its simplicity, this task usefully demonstratesthat issues with misspecification can arise even in the simplest of models.

Original: SIR.
Revised: SIR.

Original: A stochastic model of epidemic spread with a time-varying infection rate.
Revised: A stochastic model of epidemic spread with a time-varying infection rate.

Original: We attempt toinfer two parameters, the infection rate β and the recovery rate γ .
Revised: We attempt toinfer two parameters, the infection rate β and the recovery rate γ .

Original: We draw daily infection countsfor a period of 365 days and compute the following summary statistics: 1-3) Mean , Median and
Revised: We draw daily infection countsfor a period of 365 days and compute the following summary statistics: 1-3) Mean , Median and

Original: Max - the mean, median and maximum number of infections; 4) Max Day - the day of occurrenceof the maximum number of infections, 5) Half Day - the day at which half the total number ofinfections was reached, and 6) Autocor - the mean autocorrelation (lag 1) of infections.
Revised: Max - the mean, median and maximum number of infections; 4) Max Day - the day of occurrenceof the maximum number of infections, 5) Half Day - the day at which half the total number ofinfections was reached, and 6) Autocor - the mean autocorrelation (lag 1) of infections.

Original: To artificiallyproduce misspecified observations, we chose to introduce a small degree of reporting delays, wherebyweekend infection counts are reduced by 5%, which are subsequently recouped on the following
Revised: To artificiallyproduce misspecified observations, we chose to introduce a small degree of reporting delays, wherebyweekend infection counts are reduced by 5%, which are subsequently recouped on the following

Original: Monday.
Revised: Monday.

Original: Fig.
Revised: Fig.

Original: 5 in the appendix shows an example misspecified simulation and a correspondingartificially produced observation.
Revised: 5 in the appendix shows an example misspecified simulation and a correspondingartificially produced observation.

Original: CS.
Revised: CS.

Original: A marked point process model of cancer and stromal (CS) cell development in 2D-space, inwhich the locality of a cell relative to unobserved parent cells determines whether it is a cancer ora stromal cell.
Revised: A marked point process model of cancer and stromal (CS) cell development in 2D-space, inwhich the locality of a cell relative to unobserved parent cells determines whether it is a cancer ora stromal cell.

Original: Similar point process models can be found in Jones-Todd et al.
Revised: Similar point process models can be found in Jones-Todd et al.

Original: There arethree Poisson rate parameters, corresponding to the cell rate λ c , the parent rate λ p and the daughterrate
Revised: There arethree Poisson rate parameters, corresponding to the cell rate λ c , the parent rate λ p and the daughterrate

Original: λ d , where λ d controls the number of nearest cells to each parent which become cancerous.
Revised: λ d , where λ d controls the number of nearest cells to each parent which become cancerous.

Original: Weused the following summary statistics: 1-2) N Cancer and N Stromal - the number of cancer andstromal cells; 3-4) Mean Min Dist and Max Min Dist - the mean and maximum distance fromstromal cells to their nearest cancer cell.
Revised: Weused the following summary statistics: 1-2) N Cancer and N Stromal - the number of cancer andstromal cells; 3-4) Mean Min Dist and Max Min Dist - the mean and maximum distance fromstromal cells to their nearest cancer cell.

Original: Summary statistics 3-4 were approximated empiricallythrough sampling 50 stromal cells, as the full distance matrix was too expensive to compute.
Revised: Summary statistics 3-4 were approximated empiricallythrough sampling 50 stromal cells, as the full distance matrix was too expensive to compute.

Original: Toartificially produce misspecified observations, cancer cells were removed if they fell too close to thecancer parent, mimicking necrosis that often occurs in core regions of tumours.
Revised: Toartificially produce misspecified observations, cancer cells were removed if they fell too close to thecancer parent, mimicking necrosis that often occurs in core regions of tumours.

Original: A plot of an examplemisspecified simulation and a corresponding artificially produced observation can be found in Fig.
Revised: A plot of an examplemisspecified simulation and a corresponding artificially produced observation can be found in Fig.

Original: 6in the appendix.
Revised: 6in the appendix.

Original: 4.3 Performance Metrics
Revised: 4.3 Performance Metrics

Original: Log probability of θ ∗ .
Revised: Log probability of θ ∗ .

Original: The mean posterior log probability of the true parameters over multipleobservations is an extensively used performance metric (e.g. Papamakarios and Murray, 2016;
Revised: The mean posterior log probability of the true parameters over multipleobservations is an extensively used performance metric (e.g. Papamakarios and Murray, 2016;

Original: Hermans et al., 2020; Durkan et al., 2020), and is closely related to the average KL divergencebetween the true and approximate posteriors (Lueckmann et al., 2021).
Revised: Hermans et al., 2020; Durkan et al., 2020), and is closely related to the average KL divergencebetween the true and approximate posteriors (Lueckmann et al., 2021).

Original: We found that NPE wouldoccasionally fail catastrophically, placing negligible posterior mass on the true parameters, skewingthe mean estimates.
Revised: We found that NPE wouldoccasionally fail catastrophically, placing negligible posterior mass on the true parameters, skewingthe mean estimates.

Original: Due to this, we instead use box plots to visualise the distribution over thelog-probabilities of the true parameters, rather than reporting the mean.
Revised: Due to this, we instead use box plots to visualise the distribution over thelog-probabilities of the true parameters, rather than reporting the mean.

Original: Posterior Coverage.
Revised: Posterior Coverage.

Original: Given a confidence level α and a posterior approximation ˆ p ( θ | y ) , let
Revised: Given a confidence level α and a posterior approximation ˆ p ( θ | y ) , let

Original: HDR ˆ p ( θ | y ) (1 − α ) represent its 1 − α highest posterior density region, i.e. the smallest region thatcontains at least 100(1 − α )% of the mass of ˆ p ( θ | y ) .
Revised: HDR ˆ p ( θ | y ) (1 − α ) represent its 1 − α highest posterior density region, i.e. the smallest region thatcontains at least 100(1 − α )% of the mass of ˆ p ( θ | y ) .

Original: The expected posterior coverage is thefrequency with which the true parameter value falls within this highest density region
Revised: The expected posterior coverage is thefrequency with which the true parameter value falls within this highest density region

Original: [Equation]
Revised: [Equation]

Original: (10) where 1 is the indicator function.
Revised: (10) where 1 is the indicator function.

Original: Posterior coverage is useful for assessing whether an inferenceprocedure is likely to yield overconfident or conservative posterior estimates (Hermans et al., 2021).
Revised: Posterior coverage is useful for assessing whether an inferenceprocedure is likely to yield overconfident or conservative posterior estimates (Hermans et al., 2021).

Original: We empirically estimate this expectation, approximating the highest density regions using the densityquantile approach outlined in Hyndman (1996), with 10,000 samples from the approximate posterior.
Revised: We empirically estimate this expectation, approximating the highest density regions using the densityquantile approach outlined in Hyndman (1996), with 10,000 samples from the approximate posterior.

Original: 4.4 Results
Revised: 4.4 Results

Original: Overall Performance.
Revised: Overall Performance.

Original: When assessing the overall inference performance, we find that RNPEconsistently outperforms NPE across all the tasks (Fig. 2).
Revised: When assessing the overall inference performance, we find that RNPEconsistently outperforms NPE across all the tasks (Fig. 2).

Original: R NPE tends to put higher posteriordensity on the true parameters θ ∗ , frequently by several orders of magnitude (note the log scale).
Revised: R NPE tends to put higher posteriordensity on the true parameters θ ∗ , frequently by several orders of magnitude (note the log scale).

Original: Further, RNPE produces posteriors with better coverage properties, being more conservative than
Revised: Further, RNPE produces posteriors with better coverage properties, being more conservative than

Original: NPE across all the tasks (Fig. 2b).
Revised: NPE across all the tasks (Fig. 9b).

Original: N PE has very poor posterior coverage across all the tasks, withmisspecification exacerbating the issues found by Hermans et al.
Revised: N PE has very poor posterior coverage across all the tasks, withmisspecification exacerbating the issues found by Hermans et al.

Original: (2021) in the well-specified case.
Revised: (2021) in the well-specified case.

Original: Itis important to note that having conservative posterior estimates is generally considered preferable tohaving overconfident posterior estimates.
Revised: Itis important to note that having conservative posterior estimates is generally considered preferable tohaving overconfident posterior estimates.

Original: If an algorithm tends to produce overconfident posteriors, itbecomes difficult to know whether the inference should be trusted, and trusting the results may leadto drawing erroneous scientific conclusions.
Revised: If an algorithm tends to produce overconfident posteriors, itbecomes difficult to know whether the inference should be trusted, and trusting the results may leadto drawing erroneous scientific conclusions.

Original: Gaussian.
Revised: Gaussian.

Original: An example for the Gaussian task can be seen in Fig.
Revised: An example for the Gaussian task can be seen in Fig.

Original: 1, in which x 1 corresponds to themean statistic, and x 2 the variance.
Revised: 1, in which x 1 corresponds to themean statistic, and x 2 the variance.

Original: From observing the posterior misspecification probabilities and
Revised: From observing the posterior misspecification probabilities and

Original: Gaussian
Revised: Gaussian

Original: SIR
Revised: SIR

Original: CS
Revised: CS

Original: Task
Revised: Task

Original: L og - P r ob a b ilit y *
Revised: L og - P r ob a b ilit y *

Original: RNPE
Revised: RNPE

Original: NPE
Revised: NPE

Original: [Figure]
Revised: [Figure]

Original: E m p
Revised: E m p

Original: i r i ca l
Revised: i r i ca l

Original: C ov
Revised: C ov

Original: e r a g e
Revised: e r a g e

Original: Gaussian 0.
Revised: Gaussian 0.

Original: SIR 0.
Revised: SIR 0.

Original: Confidence Level
Revised: Confidence Level

Original: CS
Revised: CS

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: the denoised samples, the results clearly identify a discrepancy in which the observed variance statisticis higher than the simulated variance statistics, allowing identification of source of misspecification.
Revised: the denoised samples, the results clearly identify a discrepancy in which the observed variance statisticis higher than the simulated variance statistics, allowing identification of source of misspecification.

Original: As this is a simple example, we can analytically access the true posterior (obtained using the true datagenerating model).
Revised: As this is a simple example, we can analytically access the true posterior (obtained using the true datagenerating model).

Original: R NPE produces a posterior close to the ground-truth, whereas NPE confidentlyplaces most of the posterior mass far from the ground-truth posterior.
Revised: R NPE produces a posterior close to the ground-truth, whereas NPE confidentlyplaces most of the posterior mass far from the ground-truth posterior.

Original: SIR.
Revised: SIR.

Original: In the SIR model, the observed data were artificially corrupted by mimicking reporting delaysover the weekend.
Revised: In the SIR model, the observed data were artificially corrupted by mimicking reporting delaysover the weekend.

Original: With this in mind, we can consider how the model criticism aspect of RNPE couldfacilitate identification of the issue.
Revised: With this in mind, we can consider how the model criticism aspect of RNPE couldfacilitate identification of the issue.

Original: Fig.
Revised: Fig.

Original: 3a shows that most summary statistics are more likelyto be well-specified than was believed a priori , with the clear exception of the Autocor summarystatistic, which is inferred to be misspecified.
Revised: 3a shows that most summary statistics are more likelyto be well-specified than was believed a priori , with the clear exception of the Autocor summarystatistic, which is inferred to be misspecified.

Original: Fig.
Revised: Fig.

Original: 3b shows a density plot of the denoised data forthe Autocor summary statistic, showing that the observed autocorrelation is much lower relativeto what would be expected under the model.
Revised: 3b shows a density plot of the denoised data forthe Autocor summary statistic, showing that the observed autocorrelation is much lower relativeto what would be expected under the model.

Original: This may prompt the practitioner to investigate theday-to-day changes in infections, in which case the discontinuities introduced by the reporting delayscould be identified.
Revised: This may prompt the practitioner to investigate theday-to-day changes in infections, in which case the discontinuities introduced by the reporting delayscould be identified.

Original: Fig.
Revised: Fig.

Original: 3c shows the posteriors for RNPE and NPE .
Revised: 3c shows the posteriors for RNPE and NPE .

Original: R NPE yields a posterior with farlower variance than NPE , demonstrating an example where it does not achieve robustness simply byproducing broader posterior estimates.
Revised: R NPE yields a posterior with farlower variance than NPE , demonstrating an example where it does not achieve robustness simply byproducing broader posterior estimates.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: Autocor (a)
Revised: Autocor (a)

Original: [Figure]
Revised: [Figure]

Original: n
Revised: n

Original: [Figure]
Revised: [Figure]

Original: y (
Revised: y (

Original: [Figure]
Revised: [Figure]

Original: Simulations
Revised: Simulations

Original: [Figure]
Revised: [Figure]

Original: Observed
Revised: Observed

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: PriorNPE
Revised: PriorNPE

Original: RNPETrue
Revised: RNPETrue

Original: CS.
Revised: CS.

Original: In the CS model, the observed data was corrupted by removing cells in the core regions oftumours, mimicking necrosis (see Fig. 6).
Revised: In the CS model, the observed data was corrupted by removing cells in the core regions oftumours, mimicking necrosis (see Fig. 6).

Original: This misspecification primarily impacts the N Cancersummary statistic (the number of cancer cells).
Revised: This misspecification primarily impacts the N Cancersummary statistic (the number of cancer cells).

Original: From Figs.
Revised: From Figs.

Original: 4a,4b, we can see that the results provideevidence that the N Cancer statistic is misspecified, with the observed data having fewer cancer
Revised: 4a,4b, we can see that the results provideevidence that the N Cancer statistic is misspecified, with the observed data having fewer cancer

Original: [Figure]
Revised: [Figure]

Original: p c d c cells than would be expected if it was generated from the simulator.
Revised: p c d c cells than would be expected if it was generated from the simulator.

Original: Once again, alongside domain knowledge, this information points the practitioner to the source of misspecification (necrosis), andhence the model could be adjusted appropriately.
Revised: Once again, alongside domain knowledge, this information points the practitioner to the source of misspecification (necrosis), andhence the model could be adjusted appropriately.

Original: For this example, NPE produced an overconfidentposterior with little mass on the true parameters, whereas RNPE does not exhibit this issue (Fig. 4c).
Revised: For this example, NPE produced an overconfidentposterior with little mass on the true parameters, whereas RNPE does not exhibit this issue (Fig. 4c).

Original:  Conclusions
Revised:  Conclusions

Original: We described RNPE , an approach that can be applied to any black-box simulator to simultaneouslyperform model criticism , and perform robust inference .
Revised: We described RNPE , an approach that can be applied to any black-box simulator to simultaneouslyperform model criticism , and perform robust inference .

Original: R NPE is philosophically similar to existingrobust SBI approaches for ABC and synthetic likelihood (Wilkinson, 2013; Frazier and Drovandi,2021).
Revised: R NPE is philosophically similar to existingrobust SBI approaches for ABC and synthetic likelihood (Wilkinson, 2013; Frazier and Drovandi,2021).

Original: However, this work introduces two novel contributions: i) we propose the first strategyfor performing robust inference with a neural SBI method, which have advantages in simulationefficiency, flexibility and scaling to high dimensional problems (Lueckmann et al., 2021), and ii) weuse a spike-and-slab error model to get explicit misspecification probability diagnostics for summarystatistics, to facilitate model criticism .
Revised: However, this work introduces two novel contributions: i) we propose the first strategyfor performing robust inference with a neural SBI method, which have advantages in simulationefficiency, flexibility and scaling to high dimensional problems (Lueckmann et al., 2021), and ii) weuse a spike-and-slab error model to get explicit misspecification probability diagnostics for summarystatistics, to facilitate model criticism .

Original: Here, we restricted focus to extending NPE .
Revised: Here, we restricted focus to extending NPE .

Original: In future work itwould be interesting to develop robust extensions for other SBI algorithms and to compare results withexisting methods.
Revised: In future work itwould be interesting to develop robust extensions for other SBI algorithms and to compare results withexisting methods.

Original: Additionally, it would be beneficial to develop a larger set of realistic misspecifiedtasks for assessing performance of robust SBI methods, similar to the work by Lueckmann et al.(2021) for the well-specified case.
Revised: Additionally, it would be beneficial to develop a larger set of realistic misspecifiedtasks for assessing performance of robust SBI methods, similar to the work by Lueckmann et al.(2021) for the well-specified case.

Original: We demonstrated with three examples, that RNPE consistently outperforms NPE for parameterinference under misspecification, tending to put more posterior mass on the true parameters, whilstalso exhibiting better coverage properties.
Revised: We demonstrated with three examples, that RNPE consistently outperforms NPE for parameterinference under misspecification, tending to put more posterior mass on the true parameters, whilstalso exhibiting better coverage properties.

Original: The tasks were designed to mimic levels of misspecificationthat could be realistically encountered, and hence the poor performance of NPE calls into questionits use in practice, particularly when the level of misspecification could be significant.
Revised: The tasks were designed to mimic levels of misspecificationthat could be realistically encountered, and hence the poor performance of NPE calls into questionits use in practice, particularly when the level of misspecification could be significant.

Original: The starkdifference in performance highlights, as suggested by Box (1980), that we should not have blind faiththat a model is sufficiently accurate, and therefore we must devise methods for performing modelcriticism and robust inference .
Revised: The starkdifference in performance highlights, as suggested by Box (1980), that we should not have blind faiththat a model is sufficiently accurate, and therefore we must devise methods for performing modelcriticism and robust inference .

Original: With further research and wider adoption of these principles in SBI ,we believe simulators will only become more powerful tools for scientific discovery.
Revised: With further research and wider adoption of these principles in SBI ,we believe simulators will only become more powerful tools for scientific discovery.

Original: Checklist
Revised: Checklist

Original: For all authors...
Revised: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?
Revised: contributions and scope?

Original: [Yes]
Revised: [Yes]

Original: (b) Did you describe the limitations of your work?
Revised: (b) Did you describe the limitations of your work?

Original: [Yes] See Section 5.
Revised: [Yes] See Section 5.

Original: (c) Did you discuss any potential negative societal impacts of your work?
Revised: (c) Did you discuss any potential negative societal impacts of your work?

Original: [No] We see
Revised: [No] We see

Original: robust inference approaches allowing for better informed decision making, which
Revised: robust inference approaches allowing for better informed decision making, which

Original: should generally have positive societal consequences.
Revised: should generally have positive societal consequences.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to
Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?
Revised: them?

Original: [Yes]
Revised: [Yes]

Original: If you are including theoretical results...
Revised: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?
Revised: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you include complete proofs of all theoretical results?
Revised: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]
Revised: [N/A]

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?
Revised: mental results (either in the supplemental material or as a URL)?

Original: [Yes](b) Did you specify all the training details (e
Revised: [Yes](b) Did you specify all the training details (e

Original: g., data splits, hyperparameters, how theywere chosen)?
Revised: g., data splits, hyperparameters, how theywere chosen)?

Original: [Yes]
Revised: [Yes](c)

Original: (c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?
Revised: Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] We used a box plot (Fig. 2a) to visualise the distributionof results over seeds.
Revised: [Yes] We used a box plot (Fig. 9a) to visualise the distributionof results over seeds.

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
Revised: (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes]
Revised: [Yes]

Original: If you are using existing assets (e
Revised: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...
Revised: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?
Revised: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] NumPyro Python
Revised: [Yes] NumPyro Python

Original: package (Section 2.2).
Revised: package (Section 2.2).

Original: (b) Did you mention the license of the assets?
Revised: (b) Did you mention the license of the assets?

Original: [No]
Revised: [No]

Original: (c) Did you include any new assets either in thesupplemental
Revised: (c) Did you include any new assets either in the

Original:  material or as a URL?
Revised: supplemental material or as a URL?

Original: [No](d) Did you discuss whether
Revised: [No](d) Did you discuss whether

Original: and how consent was obtained from people whose data you’reusing/curating?
Revised: and how consent was obtained from people whose data you’reusing/curating?

Original: [N/A](e)
Revised: N/ A](e

Original: Did you discuss whetherthe data you are using/curating contains personally identifiableinformation or offensive
Revised: Did you discuss whether

Original: content?
Revised: the data

Original: N/A If you 
Revised: you are using/

Original: used
Revised: curating contains personally identifiable information or offensive

Original:  crowdsourcing or conducted research with human subjects...
Revised: content? [N/A If you used crowdsourcing or conducted research with human subjects...

Original: a) Did you include the full text of instructions given to participants and screenshots, if
Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: applicable?
Revised: applicable?

Original: [N/A
Revised: [N/A

Original: b) Did you describe any potential participant risks, with links to Institutional Review
Revised: b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board
Revised: Board

Original: (IRB) approvals, if applicable?
Revised: (IRB) approvals, if applicable?

Original: [N/A
Revised: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount
Revised: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: spent on participant compensation?
Revised: spent on participant compensation?

Original: [N/A
Revised: [N/A