Original: N EURAL A PPROXIMATE S UFFICIENT S TATISTICS FOR I MPLICIT M ODELS
Revised: N EURAL A PPROXIMATE S UFFICIENT S TATISTICS FOR I MPLICIT M ODELS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We consider the fundamental problem of how to automatically construct summary statistics for implicit generative models where the evaluation of  likelihood function is intractable but sampling/ simulating data from the model is possible.
Revised: We consider the fundamental problem of how to automatically construct summary statistics for implicit generative models where the evaluation of the likelihood function is intractable but sampling data from the model is possible.

Original: The idea is to frame the task of constructing sufﬁcient statistics as learning mutual information maximizing representation of the data. This representation is computed by a deep neural network trained by a joint statistic-posterior learning strategy.
Revised: The idea is to frame the task of constructing sufﬁcient statistics as learning mutual information maximizing representations of the data with the help of deep neural networks. The infomax learning procedure does not need to estimate any density or density ratio.

Original: We apply our approach to both traditional approximate Bayesian computation(ABC) and recent neural likelihood approaches, boosting their performance on a range of tasks.
Revised: We apply our approach to both traditional approximate Bayesian computation and recent neural likelihood methods, boosting their performance on a range of tasks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Many data generating processes can be well-described by a parametric statistical model that can be easily simulated forward but does not possess an analytical likelihood function.
Revised: Many data generating processes can be well-described by a parametric statistical model that can be easily simulated forward but does not possess an analytical likelihood function.

Original: These models are called implicit generative models (Diggle & Gratton, 1984) or simulator-based models (Lintusaari et al., 2017) and are widely used in science and engineering domains, including physics (Sjöstrand et al., 2008), genetics (Järvenpää et al., 2018), computer graphics (Mansinghka et al., 2013),  ﬁnance (Bansal & Yaron, 2004), cosmology (Weyant et al., 2013), ecology (Wood, 2010) and epidemiology (Chinazzi et al., 2020).
Revised: These models are called implicit generative models (Diggle & Gratton, 1984) or simulator-based models (Lintusaari et al., 2017) and are widely used in science and engineering domains, including physics (Sjöstrand et al., 2008), genetics (Järvenpää et al., 2018), computer graphics (Mansinghka et al., 2013), robotics (Lopez-Guevara et al., 2017), ﬁnance (Bansal & Yaron, 2004), cosmology (Weyant et al., 2013), ecology (Wood, 2010) and epidemiology (Chinazzi et al., 2020).

Original: For example, the number of infected/healthy people in an outbreak could be well modelled by stochastic differential equations (SDE) simulated by Euler-Maruyama discretization but the likelihood function of a SDE is generally non-analytical.
Revised: For example, the number of infected/healthy people in an outbreak could be well modelled by stochastic differential equations (SDE) simulated by Euler-Maruyama discretization but the likelihood function of a SDE is generally non-analytical.

Original: Directly inferring the parameters of these implicit models is often difﬁcult or even impossible.
Revised: Directly inferring the parameters of these implicit models is often very challenging.

Original: The techniques coined as likelihood-free inference open us a door for performing Bayesian inference in such circumstances.
Revised: The techniques coined as likelihood-free inference open us a door for performing Bayesian inference in such circumstances.

Original: Likelihood-free inference needs to evaluate neither the likelihood function nor its derivatives.
Revised: Likelihood-free inference needs to evaluate neither the likelihood function nor its derivatives.

Original: Rather, it only requires the ability to sample ( i.e. simulate) data from the model.
Revised: Rather, it only requires the ability to sample ( i.e. simulate) data from the model.

Original: Early approaches in approximate Bayesian computation (ABC) perform likelihood-free inference by repeatedly simulating data from the model, and pick a small subset of the simulated data close to the observed data to build the posterior (Pritchard et al., 1999;
Revised: Early approaches in approximate Bayesian computation (ABC) perform likelihood-free inference by repeatedly simulating data from the model, and pick a small subset of the simulated data close to the observed data to build the posterior (Pritchard et al., 1999;

Original: Marjoram et al., 2003; Beaumont et al., 2009; Sisson et al., 2007).
Revised: Marjoram et al., 2003; Beaumont et al., 2009; Sisson et al., 2007).

Original: Recent advances make use of ﬂexible neural density estimators to approximate either the intractable likelihood (Papamakarios et al., 2019) or directly the posterior (Papamakarios & Murray, 2016; Lueckmann et al., 2017; Greenberg et al., 2019).
Revised: Recent advances make use of ﬂexible neural density estimators to approximate either the intractable likelihood (Papamakarios et al., 2019) or directly the posterior (Papamakarios & Murray, 2016; Lueckmann et al., 2017; Greenberg et al., 2019).

Original: Despite the algorithmic differences, a shared ingredient in likelihood-free inference methods is the choice of summary statistics.
Revised: Despite the algorithmic differences, a shared ingredient in likelihood-free inference methods is the choice of summary statistics.

Original: Well-chosen summary statistics have been proven crucial for the performance of likelihood-free inference methods (Blum et al., 2013; Fearnhead & Prangle, 2012;
Revised: Well-chosen summary statistics have been proven crucial for the performance of likelihood-free inference methods (Blum et al., 2013; Fearnhead & Prangle, 2012;

Original: Sisson et al., 2018).
Revised: Sisson et al., 2018).

Original: Unfortunately, in practice it is often difﬁcult to determine low-dimensional and informative summary statistic without domain knowledge from experts.
Revised: Unfortunately, in practice it is often difﬁcult to determine low-dimensional and informative summary statistic without domain knowledge from experts.

Original: In this work, we propose a novel deep neural network-based approach for automatic construction of summary statistics.
Revised: In this work, we propose a novel deep neural network-based approach for automatic construction of summary statistics.

Original: Neural networks have been previously applied to learning summary statistics for likelihood-free inference (Jiang et al., 2017; Dinev & Gutmann, 2018; Alsing et al., 2018; Brehmer et al., 2020).
Revised: Neural networks have been previously applied to learning summary statistics for likelihood-free inference (Jiang et al., 2017; Dinev & Gutmann, 2018; Alsing et al., 2018; Brehmer et al., 2020).

Original: Our approach is unique in that our learned statistics directly targetat global sufﬁciency .
Revised: Our approach is unique in that our learned statistics directly target global sufﬁciency .

Original: The main idea is to exploit the link between statistical sufﬁciency and information theory, and to formulate the task of learning sufﬁcient statistic as the task of learning information-maximizing representation of data.
Revised: The main idea is to exploit the link between statistical sufﬁciency and information theory, and to formulate the task of learning sufﬁcient statistic as the task of learning information-maximizing representations of data.

Original: We achieve this with recent neural mutual information estimators (Belghazi et al., 2018; Hjelm et al., 2018).
Revised: We achieve this with distribution-free mutual information estimators or their proxies (Székely et al., 2014; Hjelm et al., 2018).

Original: Importantly, such neural sufﬁcient statistic could be learned jointly with the posterior, resulting in fast learning where the two could reﬁne each other iteratively.
Revised: Importantly, our statistics can be learned jointly with the posterior, resulting in fast learning where the two can reﬁne each other iteratively.

Original: To sum up, our main contributions are:
Revised: To sum up, our main contributions are:

Original: • We propose a new neural approach to automatically extract compact, near-sufﬁcient statistics from raw data.
Revised: • We propose a new neural approach to automatically extract compact , near-sufﬁcient statistics from raw data.

Original: The approach removes the need for carefully handcrafted design of summary statistics; • With the proposed statistics, we develop two new likelihood-free inference methods namely SMCABC+ and SNL+.
Revised: The approach removes the need for careful handcrafted design of summary statistics. • With the proposed statistics, we develop two new likelihood-free inference methods namely SMCABC+ and SNL +.

Original: Experiments on tasks with various types of data demonstrate their effectiveness.
Revised: Experiments on tasks with various types of data demonstrate their effectiveness.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: Likelihood-free inference .
Revised: Likelihood-free inference .

Original: LFI considers the task of Bayesian inference where the likelihood function of the model is intractable but simulating (sampling) data from the model is possible:
Revised: LFI considers the task of Bayesian inference when the likelihood function of the model is intractable but simulating (sampling) data from the model is possible:

Original: [Equation]
Revised: [Equation]

Original: where x o is the observed data, π ( θθθ ) is the prior over the model parameters θθθ , p ( x o | θθθ ) is the (possibly) non-analytical likelihood function and π ( θθθ | x o ) is the posterior over θθθ .
Revised: where x o is the observed data, π ( θθθ ) is the prior over the model parameters θθθ , p ( x o | θθθ ) is the (possibly) non-analytical likelihood function and π ( θθθ | x o ) is the posterior over θθθ .

Original: We assume  while we do not have access to the exact likelihood, we could still sample (simulate) data from the model with a simulator: x ∼ p ( x | θθθ ) .
Revised: We assume that, while we do not have access to the exact likelihood, we can still sample (simulate) data from the model with a simulator: x ∼ p ( x | θθθ ) .

Original: The task is then to infer π ( θθθ | x o ) given x o and the sampled data:
Revised: The task is then to infer π ( θθθ | x o ) given x o and the sampled data:

Original: D =
Revised: D =

Original: { θ θ θ i , x i } ni =1 where θ θ θ
Revised: { θθθ i , x i } ni =1 where θθθ i ∼ p

Original: i ∼ p ( θ θ θ ) , x i ∼ p ( x | θ θ θ i ) .
Revised: θθθ ) , x i ∼ p ( x | θθθ i ) .

Original: Note that p ( θ θ θ ) is not necessarily the prior π ( θ θ θ ) .
Revised: Note that p ( θθθ ) is not necessarily the prior π ( θθθ ) .

Original: Curse of dimensionality .
Revised: Curse of dimensionality .

Original: Different likelihood-free inference algorithms might learn π ( θθθ | x o ) in different ways, nevertheless mostof existing methods suffer from the curse of dimensionality.
Revised: Different likelihood-free inference algorithms might learn π ( θθθ | x o ) in different ways, nevertheless most existing methods suffer from the curse of dimensionality.

Original: For example, traditional ABC methods use a small subset of D closest to x o under some metric to build the posterior (Pritchard et al., 1999; Marjoram et al., 2003;
Revised: For example, traditional ABC methods use a small subset of D closest to x o under some metric to build the posterior (Pritchard et al., 1999; Marjoram et al., 2003;

Original: Beaumont et al., 2009; Sisson et al., 2007), however in high-dimensional space measuring the distance sensibly is notoriously hard (Sorzano et al., 2014; Xie et al., 2017).
Revised: Beaumont et al., 2009; Sisson et al., 2007), however in high-dimensional space measuring the distance sensibly is notoriously hard (Sorzano et al., 2014; Xie et al., 2017).

Original: On the other hand, recent advances (Papamakarios et al., 2019; Lueckmann et al., 2017; Papamakarios & Murray, 2016; Greenberg et al., 2019) utilize neural density estimators (NDE) to model the intractable likelihood or the posterior.
Revised: On the other hand, recent advances (Papamakarios et al., 2019; Lueckmann et al., 2017; Papamakarios & Murray, 2016; Greenberg et al., 2019) utilize neural density estimators (NDE) to model the intractable likelihood or the posterior.

Original: Unfortunately, modeling high-dimensional distributions with NDE accurately is also known to be very difﬁcult (Oord et al., 2016; Rippel & Adams, 2013;  Oord et al., 2017), especially when the available training(simulated) data is scarce.
Revised: Unfortunately, modeling high-dimensional distributions with NDE accurately is also known to be very difﬁcult (Rippel & Adams, 2013; Van Oord et al., 2016), especially when the available training data is scarce.

Original: Our interest here is not to design a new inference algorithm, but to ﬁnd a low-dimensional statistic s = s ( x ) that is (Bayesian) sufﬁcient:
Revised: Our interest here is not to design a new inference algorithm, but to ﬁnd a low-dimensional statistic s = s ( x ) that is (Bayesian) sufﬁcient:

Original: [Equation]
Revised: [Equation]

Original: where s : X → S is a deterministic function also learned from D .
Revised: where s : X → S is a deterministic function also learned from D .

Original: We conjecture that the learning of s ( · ) might be an easier task than direct density estimation.
Revised: We conjecture that the learning of s ( · ) might be an easier task than direct density estimation.

Original: The resultant statistic s could then be applied to a wide range of likelihood-free inference algorithms as we will elaborate in Section 3.2.
Revised: The resultant statistic s could then be applied to a wide range of likelihood-free inference algorithms as we will elaborate in Section 3.2.

Original:  M ETHODOLOGY 3. N EURAL SUFFICIENT STATISTICS
Revised:  M ETHODOLOGY 3. N EURAL SUFFICIENT STATISTICS

Original: Our new deep neural network-based approach for automatic construction of (near-)sufﬁcient statistics is based on the infomax principle, as illustrated by the following proposition :
Revised: Our new deep neural network-based approach for automatic construction of near-sufﬁcient statistics is based on the infomax principle, as illustrated by the following proposition (also see Figure 1):

Original: Proposition 1.
Revised: Proposition 1.

Original: Let θθθ ∼ p ( θθθ ) , x ∼ p ( x | θθθ ) , and s : X → S be a deterministic function.
Revised: Let θθθ ∼ p ( θθθ ) , x ∼ p ( x | θθθ ) , and s : X → S be a deterministic function.

Original: Then s = s ( x ) is a sufﬁcient statistic for p ( x | θθθ )
Revised: Then s = s ( x ) is a sufﬁcient statistic for p ( x | θθθ )

Original: if and only if
Revised: if and only if

Original: [Equation]
Revised: [Equation]

Original: where S is deterministic mapping and I ( · , · ) is the mutual information between random variables.
Revised: where S is deterministic mapping and I ( · ; · ) is the mutual information between random variables.

Original: Proof .
Revised: Proof .

Original: We defer the complete proof to the Appendix.
Revised: We defer the complete proof to the appendix.

Original: This proposition is a variant of Theorem 8 in (Shamir et al., 2008) with an adaption to  likelihood-free inference scenario.
Revised: This proposition is a variant of Theorem 8 in (Shamir et al., 2010) with an adaption to the likelihood-free inference scenario.

Original: This important result suggests that we could ﬁnd the sufﬁcient statistic s  ( x ) for a likelihood function p ( x | θθθ ) by maximizing the mutual information (MI) I ( θθθ ; S ) =
Revised: This important result suggests that we could ﬁnd the sufﬁcient statistic s = s ( x ) for a likelihood function p ( x | θθθ ) by maximizing the mutual information (MI) I ( θθθ ; s ) =

Original: KL [ p ( θθθ, S ) (cid:107) p ( θθθ ) p
Revised: KL [ p ( θθθ, s ) (cid:107) p ( θθθ ) p

Original: ( S )] between θθθ and S .
Revised: ( s )] between θθθ and s .

Original: Moreover, as our interest is in maximizing MI rather than knowing its precise value,we can maximize a non-KL surrogate, which has been proven to yield better performance (Hjelm et al., 2018; Ozair et al., 2019; Wen et al., 2020).
Revised: Moreover, as our interest is in maximizing MI rather than knowing its precise value,

Original: To this end, we utilize the Jensen-Shannon divergence (JSD)surrogate (Hjelm et al., 2018), a distribution-free, non-KLMI estimator. This estimator was shown to be much less sensitive to sample size (Hjelm et al., 2018).
Revised: Jensen-Shannon divergence (JSD) (Hjelm et al., 2018): this non-KL estimator is shown to be more robust than KL-based ones.

Original: More speciﬁcally, it estimates MI as:
Revised: More speciﬁcally, it is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where sp( t ) = log(1 + exp( t )) is the softplus function.
Revised: where sp( t ) = log(1 + exp( t )) is the softplus function.

Original: With this estimator, we set up the following objective for learning the sufﬁcient statistics, which simultaneously estimates and maximizes the MI:
Revised: With this estimator, we set up the following objective for learning the sufﬁcient statistics, which simultaneously estimates and maximizes the MI:

Original: [Equation]
Revised: [Equation]

Original: where the two deterministic mappings S and T are parameterized by two neural networks.
Revised: where the two deterministic mappings S and T are parameterized by two neural networks.

Revised: Note that we have used the law of the unconscious statistician (LOTUS) from equation 3 to equation 4.

Revised: The mini-batch version of this objective is given in the appendix.

Original: With considerable training samples and powerful neural networks S and T , we could obtain nearsufﬁcient statistic with s = arg max S max T L ( S, T ) .
Revised: Distance correlation (DC) (Székely et al., 2014): unlike the JSD estimator, this estimator does not need to learn an additional network T , and can be learned much faster. It is deﬁned as: [Equation] where

Revised: [Equation]

Revised: −

Revised: [Equation]

Revised: a

Revised: [Equation]

Revised: − b (cid:48) (cid:107) ] .

Revised: Similar to the case of the JSD estimator

Revised: [Equation]

Revised: we set up the following objective for learning the sufﬁcient statistics:

Revised: [Equation]

Revised: [Equation]

Revised: where the deterministic mapping S is parameterized by a neural network.

Revised: Again LOTUS is used from equation 5 to equation 6.

Revised: The mini-batch version of this objective is given in the appendix.

Revised: A comparison between the accuracy and efﬁciency of these two MI estimators (as well as other estimators (Belghazi et al., 2018; Ozair et al., 2019)) for infomax statistics learning is in the appendix.

Revised: With enough training samples and powerful neural networks, we can obtain near-sufﬁcient statistics with either s = arg max S max T L ( S, T ) or s = arg max S L ( S ) , depending on the estimator.

Original: The statistic s of data x is then given by
Revised: The statistic s of data x is then given by

Original: [Equation]
Revised: [Equation]

Original: In the above construction, we have not speciﬁed the form of the networks S and T .
Revised: In the above construction, we have not speciﬁed the form of the networks S and T .

Original: For T , we choose it to be a split architecture:

Original: [Equation]

Original: where T (cid:48) ( · , · ) , H ( · ) are both MLPs.

Original: For S , any prior knowledge about the data x could in principle be incorporated into its design.
Revised: For S , any prior knowledge about the data x could in principle be incorporated into its design.

Original: For example, for sequential data we can realize S as a LSTM network (Gers et al., 1999), and for exchangeable data we can realize S as a exchangeable neural network (Chan et al., 2018).
Revised: For example, for sequential data we can realize S as a transformer (Vaswani et al., 2017), and for exchangeable data we can realize S as a exchangeable neural network (Chan et al., 2018).

Original: Here we simply adopt a fully-connected architecture for S , and leave the problem-speciﬁc design of S as future work.
Revised: Here we simply adopt a fully-connected architecture for S , and leave the problem-speciﬁc design of S as future work.

Revised: For T , we choose it to be a split architecture T ( θθθ, S ( x ))

Revised: =

Revised: T (cid:48)

Revised: ( H ( θθθ ) , S ( x )) where T (cid:48) ( · , · ) , H ( · ) are both

Revised: MLPs.

Original: Therefore we separately learn  representations for x and θθθ before processing them together.
Revised: Therefore we separately learn low-dimensional representations for x and θθθ before processing them together.

Original: This could be seen as that we incorporate the inductive bias into the design of the networks that x and θθθ should not interact with each other directly, based on their true relationship, (for example, consider the  exponential family:
Revised: This could be seen as that we incorporate the inductive bias into the design of the networks that x and θθθ should not interact with each other directly, based on their true relationship (for example, consider the likelihood function of exponential family:

Original: L ( θθθ ; x ) ∝ exp ( H ( θθθ ) (cid:62) S ( x )) ).
Revised: L ( θθθ ; x ) ∝ exp ( H ( θθθ ) (cid:62) S ( x )) ).

Original: We are left with the problem of how to select d , the dimensionality of the sufﬁcient statistics.
Revised: We are left with the problem of how to select d , the dimensionality of the sufﬁcient statistics.

Original: The
Revised: The

Original: Pitman-Koopman-Darmois theorem  tells us that sufﬁcient statistics with ﬁxed dimensionality only exists for exponential family, so one might better determine d in a data-driven way.
Revised: Pitman-Koopman-Darmois theorem (Koopman, 1936) tells us that sufﬁcient statistics with ﬁxed dimensionality only exists for exponential family, so there is no universal way to select d .

Original: To do this, we can in principle set d to be as low as possible while preserving enough information about θθθ :
Revised: Here, we propose to use the following simple heuristics to determine d :

Original: [Equation]
Revised: [Equation]

Original: where s d (cid:48) is thestatistic with dimensionality d (cid:48) , d ∗ is  the dimensionality that achieves the highest MI and λ ∈ (0 , 1) quantiﬁes how much information we are to preserve in the learnt statistics. Due to space limit we do not pursue this idea further in this paper and leave its veriﬁcation to future work.
Revised: where K is the dimensionality of θθθ (which typically satisﬁes K (cid:28) D ). The rationale behind this heuristics is that the dimensionality of the sufﬁcient statistics in the exponential family is K , and exponential family has been proven reasonably accurate for posterior approximation (see e.g. Thomas et al., 2021; Pacchiardi & Dutta, 2020). By doubling the dimensionality of the statistics to 2 K we are likely to have a better representative power than the exponential family while still keeping d small.

Original: Furthermore, we have the following proposition comparing our method to the existing posteriormean-as-statistic approaches (Fearnhead & Prangle, 2012; Jiang et al., 2017).
Revised: Furthermore, we have the following proposition comparing our method to the existing posteriormean-as-statistic approaches (Fearnhead & Prangle, 2012; Jiang et al., 2017).

Original: Proposition 2.
Revised: Proposition 2.

Original: Let θθθ ∼ p ( θθθ ) and x ∼ p ( x | θθθ ) .
Revised: Let θθθ ∼ p ( θθθ ) and x ∼ p ( x | θθθ ) .

Original: Let s ( · ) be the maximizer of the following objective:
Revised: Let s ( · ) be a deterministic function that satisﬁes

Original: [Equation]
Revised: [Equation]

Original: then s = s ( x ) is generally not a maximizer of I ( S ( x ) ,θθθ ) and hence it is not a sufﬁcient statistic.
Revised: then s = s ( x ) is generally not a maximizer of I ( S ( x ); θθθ ) and hence it is not a sufﬁcient statistic.

Original: Proof .
Revised: Proof .

Original: We defer the proof to the Appendix.
Revised: We defer the proof to the appendix.

Original: This proposition tells us that unlike our method, the existing posterior-mean-as-statistic approaches widely used in likelihood-free inference community loose information about the posterior, and it is only optimal for predicting the posterior mean (Fearnhead & Prangle, 2012; Jiang et al., 2017).
Revised: This proposition tells us that unlike our method, the existing (posterior-)mean-as-statistic approaches widely used in likelihood-free inference community lose information about the posterior, and it is only optimal for predicting the posterior mean (Fearnhead & Prangle, 2012; Jiang et al., 2017).

Revised: When using this statistics in inference, it may yield inaccurate estimates of e.g. the posterior uncertainty.

Original: Nonetheless which method to use depends on the task e.g. full posteriorestimate vs. point estimate).
Revised: Nonetheless which statistics to use depends on the task, e.g. full posterior vs. point estimation.

Original: 3. D YNAMIC STATISTIC - POSTERIOR LEARNING
Revised: 3. D YNAMIC STATISTICS - POSTERIOR LEARNING

Original: The above neural sufﬁcient statistic could, in principle, be learned via a pilot run before the inference starts, as typically done in the work by Drovandi et al.
Revised: The above neural sufﬁcient statistic could, in principle, be learned via a pilot run before the inference starts, as, for example, done in the work by Drovandi et al.

Original: (2011); Fearnhead & Prangle (2012); Jiang et al.
Revised: (2011); Fearnhead & Prangle (2012); Jiang et al.

Original: Such a strategy requires extra simulation cost, and the learned statistic is kept ﬁxed during inference.
Revised: Such a strategy requires extra simulation cost, and the learned statistic is kept ﬁxed during inference.

Original: We propose a dynamic learning strategy below to overcome these limitations.
Revised: We propose a dynamic learning strategy below to overcome these limitations.

Original: Our idea is to jointly learn the statistic and the posterior in multiple rounds .
Revised: Our idea is to jointly learn the statistic and the posterior in multiple rounds .

Original: More concretely, at round r , we use the current statistic s ( · ) to build the r -th estimate to the posterior: q r ( θθθ | s o )
Revised: More concretely, at round j , we use the current statistic s ( · ) to build the j -th estimate to the posterior: q j ( θθθ | s o )

Original: ≈
Revised: ≈

Original: π ( θθθ | x o ) , 
Revised: π ( θθθ | x o ) , and at round j +1, this estimate is used as the new proposal distribution to simulate data: p j +1 ( θθθ ) ← q j ( θθθ | s o ) ,θθθ i ∼ p j +1 ( θθθ ) , x i ∼ p ( x | θθθ i ) . We then re-learn s ( · ) and q ( · ) with all the data up to the new round. In this process, s ( · ) and q ( · ) reﬁne each other: a good s ( · ) helps to learn q ( · ) more accurately, whereas an improved q ( · ) as a better proposal in turn helps to learn s ( · ) more efﬁciently.

Original: Input: prior π ( θθθ ) , observed data  x o
Revised: The theoretical basis of this multi-rounds strategy is provided by Proposition 1, which tells us that the sufﬁciency of the learned statistics is insensitive to the choice of p ( θθθ ) , the marginal distribution of θθθ in sampled data D = { x i ,θθθ i } nji =1 . This means that we are indeed safe to use any proposal distribution p l ( θθθ ) at any round l in multi-rounds learning, and in such case p ( θθθ ) after round j will be a mixture distribution formed by the proposal distributions of the previous rounds, i.e. p ( θθθ )

Original: Output: estimated posterior ˆ π ( θθθ | x o )
Revised: = 1 j (cid:80) jl = 1 p l ( θθθ ) . Input: prior π θθθ data x o Output: posterior

Original: Initialization: D= ∅ , p1 ( θθθ ) =
Revised: Initialization: D p

Original: π ( θθθ ) for j 1 to r do repeat sample θθθ( i) ∼ p
Revised: ( θθθ ) π θθθ ) r do repeat sample θθθ i ∼ p

Original: j ( θθθ ) ; simulate x( i) ∼ p ( x | θθθ i ) ;until samples
Revised: j ( θθθ ) ; simulate x i ∼ p ( x | θθθ i ) ; samples

Original: D ← D ∪ { θθθ i , x i } ni = ﬁt statistic net s ( · )with Dby equation 4 ; sort D according to (cid:107) s ( x i ) − s ( x o ) (cid:107) ; ﬁt p ( θθθ | s o ) with the top m θθθ s in D ; q j ( θθθ | s o )
Revised: D ← D ∪ { θθθ i , x i } ni = ﬁt statistic net s ( · ) D equation 4 ; sort D according to (cid:107) s ( x i ) − s ( x o ) (cid:107) ; ﬁt p ( θθθ | s o ) with the top m θθθ s in D ; q j ( θθθ | s o )

Original: ∝π ( θθθ ) / (cid:80) j pj ( θθθ  )· p ( θθθ
Revised: ∝ p ( θθθ | s o )

Original: | s o ) ; p j +1 ( θθθ )← qj ( θθθ | s o ) ; end for return 
Revised: π ( θθθ ) / jl p l ( θθθ ) ; p j +1 ( θθθ ) q ) ; end for return θ

Original: θθθ s o ) Input: prior π ( θθθ ) , observed data x o
Revised: Input o

Original: Output: estimated posterior ˆ π( θθθ | x o )
Revised: Output: estimated posterior ˆ π θθθ | x o )

Original: Initialization: D = ∅, p1 ( θθθ )= π( θθθ )for j r do repeat sample 
Revised: Initialization: D = ∅ p θθθ ) π θθθ ) do repeat sample θθθ

Original: θθθ ( i) ∼ p 
Revised: i p θθθ

Original: j ( θθθ ) ; simulate x( i) ∼ p 
Revised: ; simulate x i p samples

Original: ( x | θθθ i )
Revised: D ← D ∪ θθθ ni

Original: ; until n samples
Revised: ﬁt statistic

Original:  D ← D ∪ { θθθ i , x i } ni = ﬁt statistic net s ( · ) with Dby equation 4; convert D with the learned/selected s ( ·  ) ; 
Revised: 4; convert D with the learned s ( · ) ; ﬁt q converted D equation 11; q j θθθ s π θθθ ) · s θθθ ) ; p

Original: ﬁt q ( s | θθθ ) with converted D by equation 10; q j ( θθθ | s o ) ∝ π ( θθθ ) · q ( s o | θθθ ) ; p j +1 ( θθθ ) ← q j ( θθθ | s for θ = θ θ θ | s o ) whereas at round r +1, this estimate is used as the new proposal distribution to simulate data: p r +1 ( θθθ ) ← q r ( θθθ | s o ) ,θθθ i ∼ p r
Revised: ; end return

Original: [-+
 ( θθθ ) , x i ∼ p ( x | θθθ i ) . We then re-learn s ( · ) and q ( · ) with all the data up to the new round. In this process, s ( · ) and q ( · ) reﬁne each other: a good s ( · ) helps to learn q ( · ) more accurately, whereas an improved q ( · ) as a better proposal in turn helps to learn s ( · ) more efﬁciently.-]
Revised: [-+
 ( θθθ ) , x i ∼ p ( x | θθθ i ) . We then re-learn s ( · ) and q ( · ) with all the data up to the new round. In this process, s ( · ) and q ( · ) reﬁne each other: a good s ( · ) helps to learn q ( · ) more accurately, whereas an improved q ( · ) as a better proposal in turn helps to learn s ( · ) more efﬁciently.-][Equation]

Original: In practice, any likelihood-free inference algorithm that learns the posterior sequentially naturally ﬁts well within the above joint statistic-posterior learning framework.
Revised: In practice, any likelihood-free inference algorithm that learns the posterior sequentially naturally ﬁts well within the above joint statistic-posterior learning strategy.

Original: Here we study two such instances:
Revised: Here we study two such instances:

Original: Sequential Monte Carlo ABC (SMC-ABC) (Beaumont et al., 2009).
Revised: Sequential Monte Carlo ABC (SMC-ABC) (Beaumont et al., 2009).

Original: This classical algorithm learns the posterior in a non-parametric way within multiple rounds.
Revised: This classical algorithm learns the posterior in a non-parametric way within multiple rounds.

Original: Here, we consider a variant of it to better make use of the above neural sufﬁcient statistic, and to re-use all previous simulated data.
Revised: Here, we consider a variant of it to better make use of the above neural sufﬁcient statistic, and to re-use all previous simulated data.

Original: The new SMC-ABC algorithm estimates the posterior q r ( θθθ | s o ) at the r -th round as follows.
Revised: The new SMC-ABC algorithm estimates the posterior q j ( θθθ | s o ) at the j -th round as follows.

Original: We ﬁrst sort data in D
Revised: We ﬁrst sort data in D

Original: = { x i ,θθθ i } nri =1 according to the distances (cid:107) s ( x i ) − s ( x o ) (cid:107) .
Revised: = { x i ,θθθ i } nji =1 according to the distances (cid:107) s ( x i ) − s ( x o ) (cid:107) .

Original: We then pick the top- m θθθ s whose corresponding distances are the smallest.
Revised: We then pick the top- m θθθ s whose corresponding distances are the smallest.

Original: The picked θθθ s then follow θθθ ∼ p ( θθθ | s o ) as below:
Revised: The picked θθθ s then follow θθθ ∼ p ( θθθ | s o ) as below:

Original: [Equation]
Revised: [Equation]

Original: where the threshold (cid:15) is implicitly deﬁned by the ratio mnr (which automatically goes to zero as r → ∞ ).
Revised: where the threshold (cid:15) is implicitly deﬁned by the ratio mnj (which automatically goes to zero as j → ∞ ).

Original: We then ﬁt p ( θθθ | s o ) with the collected θθθ s by a ﬂexible parametric model (e.g. a Gaussian copula), with which we can obtain the r -th estimate to the posterior by importance (re-)weighting:
Revised: We then ﬁt p ( θθθ | s o ) with the collected θθθ s by a ﬂexible parametric model (e.g. a Gaussian copula), with which we can obtain the j -th estimate to the posterior by importance (re-)weighting:

Original: [Equation]
Revised: [Equation]

Original: The whole procedure of the new inference algorithm, SMC-ABC+, is summarized in Algorithm 1.
Revised: The whole procedure of this new inference algorithm, SMC-ABC+, is summarized in Algorithm 1.

Original: Sequential Neural Likelihood (SNL) (Papamakarios et al., 2019).
Revised: Sequential Neural Likelihood (SNL) (Papamakarios et al., 2019).

Original: This recent algorithm learns the posterior in a parametric way, also in multiple rounds.
Revised: This recent algorithm learns the posterior in a parametric way, also in multiple rounds.

Original: The original SNL method approximates the likelihood function p ( x | θθθ ) by a conditional neural density estimator q ( x | θθθ ) , which could be difﬁcult to learn if the dimensionality of x is high.
Revised: The original SNL method approximates the likelihood function p ( x | θθθ ) by a conditional neural density estimator q ( x | θθθ ) , which could be difﬁcult to learn if the dimensionality of x is high.

Original: Here, we alleviate such difﬁculty with our neural statistic.
Revised: Here, we alleviate such difﬁculty with our neural statistic.

Original: The new SNL algorithm estimates the posterior q r ( θθθ | s o ) at the r -th round as follows.
Revised: The new SNL algorithm estimates the posterior q j ( θθθ | s o ) at the j -th round as follows.

Original: At round r , where we have nr simulated data D = { θθθ i , x i } nri =1 , we ﬁt a neural density estimator q ( s | θθθ ) as:
Revised: At round j , where we have nj simulated data D = { θθθ i , x i } nji =1 , we ﬁt a neural density estimator q ( s | θθθ ) as:

Original: [Equation]
Revised: [Equation]

Original: where s ( · ) is the current statistic network.
Revised: where s ( · ) is the current statistic network and Q is a neural density estimator (e.g. Durkan et al. [Equation] ; Papamakarios et al.

Original: With nr being moderately large, this would yield us q ( s | θθθ )
Revised: With nj being moderately large and Q ﬂexible enough, this would yield us q ( s | θθθ )

Original: ≈ p
Revised: ≈ p

Original: ( s | θθθ ) .
Revised: ( s | θθθ ) .

Original: We then obtain the r -th estimate of the posterior by Bayes rule:
Revised: We then obtain the j -th estimate of the posterior by Bayes rule:

Original: [Equation]
Revised: [Equation]

Original: The whole procedure of this new SNL algorithm, denoted as SNL+, is summarized in Algorithm 2.
Revised: The whole procedure of this new SNL algorithm, denoted as SNL+, is summarized in Algorithm 2.

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: Approximate Bayesian computation .
Revised: Approximate Bayesian computation .

Original: ABC denotes techniques for likelihood-free inference which work by repeatedly simulating data from the model and picking thosedata similar to the observed data to estimate the posterior (Sisson et al., 2018).
Revised: ABC denotes techniques for likelihood-free inference which work by repeatedly simulating data from the model and picking those similar to the observed data to estimate the posterior (Sisson et al., 2018).

Original: Naive ABC performs simulation with the prior, whereasadvanced variants like MCMC-ABC (Marjoram et al., 2003; Meeds et al., 2015) and SMC-ABC (Beaumont et al., 2009; Sisson et al., 2007) perform simulation with informed proposals.
Revised: Naive ABC performs simulation with the prior, whereas MCMC-ABC (Marjoram et al., 2003; Meeds et al., 2015) and SMC-ABC (Beaumont et al., 2009; Sisson et al., 2007) use informed proposals, and more advanced methods employ experimental design or active learning to accelerate the inference (Gutmann & Corander, 2016; Järvenpää et al., 2019).

Original: To measure the similarity to the observed data sensibly, it is often wise to use low-dimensional summary statistic rather than the raw data in ABC.
Revised: To measure the similarity to the observed data, it is often wise to use low-dimensional summary statistics rather than the raw data.

Original: Here we develop a way to learn compact sufﬁcient statistic for ABC.
Revised: Here we develop a way to learn compact sufﬁcient statistics for ABC.

Original: Neural density estimator-based inference .
Revised: Neural density estimator-based inference .

Original: Apart from ABC, a recent line of research uses a conditional neural density estimator to (sequentially) learn the intractable likelihood (e.g SNL Papamakarios et al.
Revised: Apart from ABC, a recent line of research uses a conditional neural density estimator to (sequentially) learn the intractable likelihood (e.g SNL Papamakarios et al.

Original: ; Lueckmann et al.
Revised: ; Lueckmann et al.

Original: or directly the posterior (e.g SNPE Papamakarios &
Revised: or directly the posterior (e.g SNPE Papamakarios &

Original: Murray (2016); Lueckmann et al. (2017); Greenberg et al.
Revised: Murray (2016); Lueckmann et al. (2017); Greenberg et al.

Original: Likelihood-targeting approaches has the advantage that they could readily make use of any proposal distribution in sequential learning, but  rely on low-dimensional, well-chosen summary statistic.
Revised: Likelihood-targeting approaches have the advantage that they could readily make use of any proposal distribution in sequential learning, but they rely on low-dimensional, well-chosen summary statistic.

Original: Posterior-targeting methods on the contrary need no design of summary statistic, but they require non-trivial efforts to facilitate sequential learning.
Revised: Posterior-targeting methods on the contrary need no design of summary statistic, but they require non-trivial efforts to facilitate sequential learning.

Original: Our approach (e.g SNL+) could be seen as taking the advantages from both worlds.
Revised: Our approach (e.g SNL+) can be seen as taking the advantages from both worlds.

Original: Automatic construction of summary statistics .
Revised: Automatic construction of summary statistics .

Original: A set of works have been proposed to automatically construct low-dimensional summary statistics.
Revised: A set of works have been proposed to automatically construct low-dimensional summary statistics.

Original: Two lines of them are most related to our approach.
Revised: Two lines of them are most related to our approach.

Original: The ﬁrst line (Fearnhead & Prangle, 2012) and its variants (Jiang et al., 2017; Chan et al., 2018; 
Revised: The ﬁrst line (Fearnhead & Prangle, 2012; Jiang et al., 2017; Chan et al., 2018; Wiqvist et al., 2019;

Original: Wiqvist et al., 2019; Dinev & Gutmann, 2018) train a neural network to predict the posterior mean and use this prediction as the summary statistic.
Revised: Dinev & Gutmann, 2018) train a neural network to predict the posterior mean and use this prediction as the summary statistic.

Original: These mean-as-statistic approaches, as analyzed previously in Proposition 2, indeed do not guarantee sufﬁciency.
Revised: These mean-as-statistic approaches, as analyzed previously in Proposition 2, indeed do not guarantee sufﬁciency.

Original: Rather than taking the predicted mean, the works (Alsing et al., 2018; Brehmer et al., 2020) take the score function ∇ θθθ log p ( x | θθθ ) | θθθ = θθθ ∗ around some ﬁducial parameter θθθ ∗ as the summary statistic.
Revised: Rather than taking the predicted mean, the works (Alsing et al., 2018; Brehmer et al., 2020) take the score function ∇ θθθ log p ( x | θθθ ) | θθθ = θθθ ∗ around some ﬁducial parameter θθθ ∗ as the summary statistic.

Original: However, these score-as-statistic approaches are only locally sufﬁcient around θθθ ∗ , and it requires to ﬁrst identify θθθ ∗ which could itself be difﬁcult.
Revised: However, these score-as-statistic methods are only locally sufﬁcient around θθθ ∗ .

Original: Our approach differs from all these methods as it is globally sufﬁcient for all θθθ .
Revised: Our approach differs from all these methods as it is globally sufﬁcient for all θθθ .

Original: MI and ratio estimation .
Revised: MI and ratio estimation .

Original: It has been shown in the literature that many variational MI estimators
Revised: It has been shown in the literature that many variational MI estimators

Original: I ( X ; Y ) also estimate the ratio p ( X, Y ) /p
Revised: I ( X ; Y ) also estimate the ratio p ( X, Y ) /p

Original: ( X ) p ( Y ) up to a constant (Nowozin et al., 2016; Nguyen et al., 2010).
Revised: ( X ) p ( Y ) up to a constant (Nowozin et al., 2016; Nguyen et al., 2010).

Original: Therefore our MI-based statistic learning method is closely related to ratio estimation approaches like (Cranmer et al., 2015; Thomas et al., 2016; Hermans et al., 2019).
Revised: Therefore our MI-based statistic learning method is closely related to ratio estimating approaches to posterior inference (Hermans et al., 2020; Thomas et al., 2021).

Original: The differences are 1) weestimate the ratio in the low-dimensional space rather than in the original space by a particular network architecture, based on a novel sufﬁcient statistics perspective; 2) we decouple the task of statistic learning from the task of density estimation, which grants us the privilege to use any infomax representation learning methods even if they are ratio-free, e.g. Ozair et al. Wen et al.
Revised: The differences are 1) we decouple the task of statistics learning from the task of density estimation for LFI, which grants us the privilege to use any infomax representation learning methods that are ratio-free (Székely et al., 2014; Ozair et al., 2019); and 2) even if we do estimate the ratio, we do this in the low-dimensional space based on a sufﬁcient statistics perspective, which is typically easier than in the original space.

Original:  E XPERIMENTS 5. S ETUP
Revised:  E XPERIMENTS 5. S ETUP

Original: Baselines .
Revised: Baselines .

Original: We apply the proposed statistic to two aforementioned likelihood-free inference methods: (i) SMC-ABC (Beaumont et al., 2009)(the slightly modiﬁed version) and (ii) SNL (Papamakarios et al., 2019).
Revised: We apply the proposed statistics to two aforementioned likelihood-free inference methods: (i) SMC-ABC (Beaumont et al., 2009) and (ii) SNL (Papamakarios et al., 2019).

Original: We compare the performance of the algorithms augmented with our neural statistic (dubbed as SMC-ABC+ and SNL+) to their original versions as well as the versions based on expertdesigned statistics (details presented later; we call the corresponding methods SMC-ABC’ and SNL’).
Revised: We compare the performance of the algorithms augmented with our neural statistics (dubbed as SMC-ABC+ and SNL+) to their original versions as well as the versions based on expert-designed statistics (details presented later; we call the corresponding methods SMC-ABC’ and SNL’).

Original: We also compare to the sequential neural posterior estimate (SNPE) method 1 which needs no statistic design, as well as the sequential ratio estimate (SRE) method (Hermans et al., 2019) which is closely related to our MI-based method 2 .
Revised: We also compare to the sequential neural posterior estimate (SNPE) method 1 which needs no statistic design, as well as the sequential ratio estimate (SRE) method (Hermans et al., 2020) which is closely related to our MI-based method 2 .

Original: All methods are run for 10 rounds with 1,000 simulations each.
Revised: All methods are run for 10 rounds with 1,000 simulations each.

Revised: The results presented below are for the JSD estimator; the DC estimator achieves similar accuracy (see appendix).

Original: Evaluation metric .
Revised: Evaluation metric .

Original: To assess the quality of the estimated posterior, we compare the Jensen-Shannon divergence (JSD) between the approximate posterior Q and the true posterior P in each method: [Equation]
Revised: To assess the quality of the estimated posterior, we compare the Jensen-Shannon divergence (JSD) between the approximate posterior Q and the true posterior P for each method

Original: [Figure]
Revised: [Figure]

Original: 10 rounds 0.
Revised: 10 rounds 0.

Original: J S D
Revised: J S D

Original: SMC-ABC
Revised: SMC-ABC

Original: SMC-ABC+
Revised: SMC-ABC+

Original: SNL
Revised: SNL

Original: SNL+ (b)(c)
Revised: SNL+ (b)

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Original: [Table]
Revised: [Table]

Original:  For the problems we consider in the experiments, the true posterior P is either analytically available, or could be accurately approximated by a standard rejection ABC algorithm (Pritchard et al., 1999) with known low-dimensional sufﬁcient statistic (e.g s ( x ) ∈ Z ) and extensive simulations (e.g 10 6 ).
Revised: (see appendix). For the problems we consider, the true posterior P is either analytically available, or can be accurately approximated by a standard rejection ABC algorithm (Pritchard et al., 1999) with known low-dimensional sufﬁcient statistic (e.g s ( x ) ∈ Z ) and extensive simulations (e.g 10 6 ).

Original: 5. R ESULTS
Revised: 5. R ESULTS

Original: We demonstrate the effectiveness of our method in three cases: (a) an Ising model; (b) a Gaussian copula model; (c) an Ornstein-Uhlenbeck process.
Revised: We demonstrate the effectiveness of our method on three models: (a) an Ising model; (b) a Gaussian copula model; (c) an Ornstein-Uhlenbeck process.

Original: The Ising model does not have an analytical likelihood but the posterior can be approximated accurately by rejection ABC due to the existence of low-dimensional, discrete sufﬁcient statistic.
Revised: The Ising model does not have an analytical likelihood but the posterior can be approximated accurately by rejection ABC due to the existence of low-dimensional, discrete sufﬁcient statistic.

Original: The last two models have analytical likelihoods and hence analytical posteriors.
Revised: The last two models have analytical likelihoods and hence analytical posteriors.

Original: These models cover the cases of graph data, i.i.d data and sequence data.
Revised: These models cover the cases of graph data, i.i.d data and sequence data.

Original: Ising model .
Revised: Ising model .

Original: The ﬁrst model we consider is a mathematical model in statistical physics that describes the states of atomic spins on a 8 × 8 lattice (see Figure 1(a)).
Revised: The ﬁrst model we consider is a mathematical model in statistical physics that describes the states of atomic spins on a 8 × 8 lattice (see Figure 1(a)).

Original: Each spin has two states described by a discrete random variable x i ∈ {− 1 , +1 } , and is only allowed to interact with its neighbour.
Revised: Each spin has two states described by a discrete random variable x i ∈ {− 1 , +1 } , and is only allowed to interact with its neighbour.

Original: Given parameters θθθ
Revised: Given parameters θθθ

Original: = { θ 1 , θ 2 }
Revised: = { θ 1 , θ 2 }

Original: , the probability density function of the Ising model is:
Revised: , the probability density function of the Ising model is:

Original: [Equation]
Revised: [Equation]

Original: where (cid:104) i, j (cid:105) denotes that spin i and spin j are neighbours.
Revised: where (cid:104) i, j (cid:105) denotes that spin i and spin j are neighbours.

Original: H is also called the Hamiltonian of the model.
Revised: H is also called the Hamiltonian of the model.

Original: Here, the likelihood function of this model is not analytical due to the intractable normalizing constant Z ( θθθ ) =
Revised: The likelihood function of this model is not analytical due to the intractable normalizing constant Z ( θθθ ) =

Original: (cid:80) x ∈{− 1 , 1 } m · m e − H ( x ; θθθ ) .
Revised: (cid:80) x ∈{− 1 , 1 } m · m exp[ − H ( x ; θθθ )] .

Original: However, sampling from the model by MCMC is possible.
Revised: However, sampling from the model by MCMC is possible.

Original: Note that the sufﬁcient statistic is known for this model: s ∗ ( x ) = { (cid:80) (cid:104) i,j (cid:105) x i x j , (cid:80) i x i } .
Revised: Note that the sufﬁcient statistics are known for this model: s ∗ ( x ) = { (cid:80) (cid:104) i,j (cid:105) x i x j , (cid:80) i x i } .

Original: The true posteriorin this model can easily be approximated bya rejection ABCalgorithm run with this low-dimensional sufﬁcient statistic and extensive simulations.
Revised: The true posterior can easily be approximated by rejection ABC with the low-dimensional sufﬁcient statistics and extensive simulations.

Original: Here, we assume that θ 2 is known, and the task is to infer the posterior of θ 1 under an uniform prior θ 1 ∼ U (0 , 1 . 5) (in this case the sufﬁcient statistic becomes only 1D: s ∗ ( x )
Revised: Here, we assume that θ 2 is known, and the task is to infer the posterior of θ 1 under an uniform prior θ 1 ∼ U (0 , 1 . 5) (in this case the sufﬁcient statistic becomes only 1D: s ∗ ( x )

Original: = (cid:80) (cid:104) i,j (cid:105) x i x j ).
Revised: = (cid:80) (cid:104) i,j (cid:105) x i x j ).

Original: The true parameters are θθθ ∗ = { 0 . 3 , 0 . 1 } .
Revised: The true parameters are θθθ ∗ = { 0 . 3 , 0 . 1 } .

Original: In Figure 1(c), we investigate whether the proposed statistic could achieve sufﬁciency.
Revised: In Figure 1(c), we investigate whether the proposed statistic could achieve sufﬁciency.

Original: Ideally, if the learned statistic s in our method does recover the true sufﬁcient statistic s ∗ well, the relationship between s and s ∗ should be nearly this
Revised: Ideally, if the learned statistic s in our method does recover the true sufﬁcient statistic s ∗ well, the relationship between s and s ∗ should be nearly monotonic (

Original: [Figure]
Revised: [Figure]

Original:     10 rounds 0.
Revised: )    10 rounds 0.

Original: J S D
Revised: J S D

Original: SMC-ABC
Revised: SMC-ABC

Original: SMC-ABC+
Revised: SMC-ABC+

Original: SNL
Revised: SNL

Original: SNL+ (b)
Revised: SNL+ (b)

Original: Truth
Revised: Truth

Original: SMC
Revised: SMC

Original: SMC+
Revised: SMC+

Original: SNL
Revised: SNL

Original: SNL+ (c)
Revised: SNL+ (c)

Original: [Figure]

Original: [Table]
Revised: [Table]

Original: Here SMC’ and SNL’ utilize the hand-crafted summary statistics guided by human prior knowledge.
Revised: Here SMC’ and SNL’ utilize the hand-crafted summary statistics guided by human prior knowledge.

Original: we plot the relationship between s ∗ and s .
Revised: we plot the relationship between s ∗ and s .

Original: We see from the ﬁgure that s learned in our method does increase monotonically with s ∗ approximately, suggesting that s recovers s ∗ reasonably well.
Revised: We see from the ﬁgure that s learned by our method does, approximately, increase monotonically with s ∗ , suggesting that s recovers s ∗ reasonably well.

Original: As comparison, the statistic learned in the widely-used posterior-mean-as-statistic approach only has weak dependence on the true sufﬁcient statistic; it is nearly indistinguishable for different s ∗ .
Revised: In comparison, the statistics learned with the widely-used posterior-mean-as-statistics approach only weakly depends on the true sufﬁcient statistic; it is nearly indistinguishable for different s ∗ .

Original: In other words, it loses sufﬁciency.
Revised: In other words, it loses sufﬁciency.

Original: The result also veriﬁes ourprevious theoretical result in Proposition 2.
Revised: The result empirically veriﬁes our theoretical result in Proposition 2.

Original: Figure 1(b) further shows the JSD between the true and learned posterior for different methods across the rounds (the vertical lines indicates standard errors, each JSD is obtained by calculating the average of 3 independent runs.
Revised: Figure 1(b) further shows the JSD between the true and learned posterior for different methods across the rounds (the vertical lines indicates standard derivation, each JSD is obtained by calculating the average of 5 independent runs.

Original: The results shown in the below experiments have the same setup).
Revised: The results shown in the below experiments have the same setup).

Original: It can be seen from the ﬁgure that for this model, likelihood-free inference methods augmented with the proposed statistic (SMC-ABC+, SNL+) outperform their original counterparts (SMC-ABC, SNL) by a large margin.
Revised: It can be seen from the ﬁgure that for this model, likelihood-free inference methods augmented with the proposed statistic (SMC-ABC+, SNL+) outperform their original counterparts (SMC-ABC, SNL) by a large margin.

Original: In Table 1, we further compare our statistic with the expert designed statistic, from which one can see their close performance (here the expert statistic is taken as the true sufﬁcient statistic s ∗ ).
Revised: In Table 1, we further compare our statistics with the expert designed statistics, from which one can see their close performance (here the expert statistics is taken as the true sufﬁcient statistics s ∗ ).

Original: We also see that our method also seems to outperform SRE which directly estimates the ratio t ( x ,θθθ )
Revised: We further see that our method also outperforms SRE which directly estimates the ratio t ( x ,θθθ )

Original: =
Revised: =

Original: p ( x ,θθθ ) /p
Revised: p ( x ,θθθ ) /p

Original: ( x ) p ( θθθ ) ∝ L ( θθθ ; x ) in high-dimensional space (note that the true likelihood isactually of the form L ( θθθ ; x ) =
Revised: ( x ) p ( θθθ ) ∝ L ( θθθ ; x ) in high-dimensional space (note that the true likelihood is of the form L ( θθθ ; x ) =

Original: exp( θθθs ∗ ( x )) /Z
Revised: exp( θθθs ∗ ( x )) /Z

Original: ( θθθ ) ) as well as SNPE (version B).
Revised: ( θθθ ) ) as well as SNPE (version B).

Original: The reason why SNPE(-B) does not perform more satisfactorily might bedue to the fact that it relies on importance weights to facilitate sequential learning, which induces high variance that makes its training unstable.
Revised: The reason why SNPE(-B) does not perform more satisfactorily might be that it relies on importance weights to facilitate sequential learning, which can induce high variance that makes the training unstable.

Original: Gaussian copula .
Revised: Gaussian copula .

Original: The second model we consider is a 2D Gaussian copula model (Chen & Gutmann, 2019).
Revised: The second model we consider is a 2D Gaussian copula model (Chen & Gutmann, 2019).

Original: Data x in this model can beseen as generated from the latent variable z as follows:
Revised: Data x for this model can be generated with aid of a latent variable z as follows:

Original: [Equation]
Revised: [Equation]

Original: where Φ( · ) , F 1 ( x 1 ; θ 1 ) , F 2 ( x 2 ; θ 2 ) are the cumulative distribution function (CDF) of  standard normal distribution, the CDF of f 1 ( x 1 ; θ 1 ) and the CDF of f 2 ( x 2 ; θ 2 ) respectively.
Revised: where Φ( · ) , F 1 ( x 1 ; θ 1 ) , F 2 ( x 2 ; θ 2 ) are the cumulative distribution function (CDF) of the standard normal distribution, the CDF of f 1 ( x 1 ; θ 1 ) and the CDF of f 2 ( x 2 ; θ 2 ) respectively.

Original: We assume that a total number of 200 samples are i.i.d drawn from this model, yielding a population X
Revised: We assume that a total number of 200 samples are i.i.d drawn from this model, yielding a population X

Original: = { x i } 200 i =1 that serves as our observed data.
Revised: = { x i } 200 i = that serves as our observed data.

Original: Note that the likelihood of this model can be computed analytically by the law of variable transformation.
Revised: Note that the likelihood of this model can be computed analytically by the law of variable transformation.

Original: To perform inference, we compute a rudimentary statistic to describe X , namely (a) the 20-equally spaced quantiles of the marginal distributions of X and (b) the correlation between the latent variables z 1 , z 2 in X , resulting in a statistic of dimensionality 41.
Revised: To perform inference, we compute a rudimentary statistic to describe X , namely (a) the 20-equally spaced quantiles of the marginal distributions of X and (b) the correlation between the latent variables z 1 , z 2 in X , resulting in a statistic of dimensionality 41.

Original: An uniform prior is set: θ 1 ∼ U (0 . 5 , 12 . 5) , θ 2 ∼ U (0 , 1) , θ 3 ∼ U (0 . 4 , 0 .
Revised: A uniform prior is used: θ 1 ∼ U (0 . 5 , 12 . 5) , θ 2 ∼ U (0 , 1) , θ 3 ∼ U (0 . 4 , 0 .

Original: 8) and θθθ ∗ = { 6 , 0 . 5 , 0 . 6 } .
Revised: 8) and θθθ ∗ = { 6 , 0 . 5 , 0 . 6 } .

Original: In Figure 2(b), we demonstrate the power of our neural sufﬁcient statistic learning method on the
Revised: In Figure 2(b), we demonstrate the power of our neural sufﬁcient statistic learning method on the

Original: Gaussian copula problem.
Revised: Gaussian copula problem.

Original: Overall, we see that the proposed method improves the accuracy of
Revised: Overall, we see that the proposed method improves the accuracy of

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: 10 rounds 0.

Original: J S D
Revised: J S D

Original: SMC-ABC
Revised: SMC-ABC

Original: SMC-ABC+
Revised: SMC-ABC+

Original: SNL
Revised: SNL

Original: SNL+ (b)
Revised: SNL+ (b)

Original: Truth
Revised: Truth

Original: SMC
Revised: SMC

Original: SMC+
Revised: SMC+

Original: SNL
Revised: SNL

Original: SNL+ (c)
Revised: SNL+ (c)

Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: Here SMC’ and SNL’ utilize the hand-crafted summary statistics guided by human prior knowledge.
Revised: [Figure] [Table] [Figure] [Table]

Original: existing likelihood-free inference methods, as well as their robustness (see e.g the reduced variability in SNL+.
Revised: [Figure] [Table] [Figure] [Table]

Original: The high variability in SNL may be due to the lack of training data required to learn the 41-dimensional likelihood function well).
Revised: [Figure] [Table] [Figure] Here SMC’ and SNL’ utilize the hand crafted summary statistics by existing likelihood-free inference methods, as well as their robustness, see e.g. the reduced variability for SNL+ (the high variability in SNL may be due to the lack of training data required to learn the 41-dimensional likelihood function well).

Original: This is also conﬁrmed by the contours plots in Figure 2(c).
Revised: This is also conﬁrmed by the contours plots in Figure 2(c).

Original: In Table 2 we further compare the proposed statistic with the expert-designed low-dimension statistic (here the expert statistic is taken to be the 5 -equally spaced marginal quantiles + the correlations between z 1 , z 2 ), from which we see that our proposed statistic achieves a better performance.
Revised: In Table 2 we further compare the proposed statistic with the expert-designed low-dimension statistic (here the expert statistic is taken to be the 5 -equally spaced marginal quantiles and the correlations between z 1 , z 2 ), from which we see that our proposed statistic achieves a better performance.

Original: For this model,  our method seems to perform slightly worse than SNPE (possibly due to the imperfect infomax learning on this problem), but the gap there is indeed very small ( ≤ 0 .
Revised: For this model, the average performance of our method is slightly worse than that of SNPE.

Original: 005 JSD).
Revised: However,

Original: We conjecture that more reliable infomax learning methods might help to resolve this issue.
Revised: SNPE has a higher variability, so that the difference in performance is actually not signiﬁcant.

Original: Ornstein-Uhlenbeck process .
Revised: Ornstein-Uhlenbeck process .

Original: The last model we consider is adiscretized stochastic differential equation (SDE).
Revised: The last model we consider is a stochastic differential equation (SDE).

Original: Data x = { x t } Dt =1 in this model is sequentially generated as:
Revised: Data x = { x t } Dt =1 in this model is sequentially generated as:

Original: [Equation]
Revised: [Equation]

Original: where D = 50 , ∆ t = 0 .
Revised: where D = 50 , ∆ t = 0 .

Original: 2 and x 0 = 10 .
Revised: 2 and x 0 = 10 .

Original: This model is Markovian, and has an analytical likelihood.
Revised: This SDE can be simulated by the Euler-Maruyama method, and has an analytical likelihood.

Original: It has a wide applications in ﬁnancial mathematics and  physical sciences.
Revised: It has a wide application in ﬁnancial mathematics and the physical sciences.

Original: Here, the parameters of interest are θθθ = { θ 1 , θ 2 } , and a uniform prior is placed on these parameters:
Revised: Here, the parameters of interest are θθθ = { θ 1 , θ 2 } , and a uniform prior is placed on them:

Original: θ 1 ∼ U (0 , 1) , θ 1 ∼ U ( − 2 . 0 , 2 . 0) .
Revised: θ 1 ∼ U (0 , 1) , θ 1 ∼ U ( − 2 . 0 , 2 . 0) .

Original: The true parameters are set to be θθθ ∗ = { 0 . 5 , 1 . 0 } .
Revised: The true parameters are set to be θθθ ∗ = { 0 . 5 , 1 . 0 } .

Original: Figure 3(b) compares the JSD of each method against the simulation cost.
Revised: Figure 3(b) compares the JSD of each method against the simulation cost.

Original: Again, we ﬁnd that the proposed neural sufﬁcient statistics greatly improves the performance of both SMC-ABC and SNL.
Revised: Again, we ﬁnd that the proposed neural sufﬁcient statistics greatly improve the performance of both SMC-ABC and SNL.

Original: In
Revised: In

Original: Table 3, we compare our statistics to expert statistic (here the expert statistics are taken as the mean, standard error and autocorrelation with lag 1, 2, 3 of the time series).
Revised: Table 3, we compare our statistics to expert statistics (here the expert statistics are taken as the mean, standard error and autocorrelation with lag 1, 2, 3 of the time series).

Original: It can be seen that our statistics are comparable to the expert statistics.
Revised: It can be seen that our statistics perform comparably to the expert statistics.

Original: Our method also seems to outperform SRE and SNPE.
Revised: Our method also seems to outperform SRE and SNPE.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We propose a new deep learning-based approach for automatically constructing low-dimensional sufﬁcient statistics in likelihood-free inference.
Revised: We proposed a new deep learning-based approach for automatically constructing low-dimensional sufﬁcient statistics for likelihood-free inference.

Original: The obtained neural approximate sufﬁcient statistics can be applied to both traditional ABC-based and recent NDE-based methods.
Revised: The obtained neural approximate sufﬁcient statistics can be applied to both traditional ABC-based and recent NDE-based methods.

Original: The main hypothesisof the approach is that learning such sufﬁcient statistic via the infomax principle might be easier than estimating the density itself.
Revised: Our main hypothesis is that learning such sufﬁcient statistics via the infomax principle might be easier than estimating the density itself.

Original: We verify this hypothesis by experiments on various tasks with graphs, i.i.d and sequence data.
Revised: We verify this hypothesis by experiments on various tasks with graphs, i.i.d and sequence data.

Original: Our method establishes a link between representation learning and likelihoodfree inference communities.
Revised: Our method establishes a link between representation learning and likelihood-free inference communities.

Original: For future works, we can consider other infomax representation learning approaches, as well as  ways toautomatically determine the dimensionality 
Revised: For future works, we can consider further infomax representation learning approaches, as well as more principle ways to determine the dimensionality of the sufﬁcient statistics.

Original: A T HEORETICAL PROOFS A. P ROOF OF P ROPOSITION
Revised: A T HEORETICAL PROOFS A. P ROOF OF P ROPOSITION

Original: Proof .
Revised: Proof .

Original: Firstly, assume s ( · ) is a sufﬁcient statistic.
Revised: Firstly, assume s ( · ) is a sufﬁcient statistic.

Original: By the deﬁnition of sufﬁcient statistic we know p ( x | θθθ ) =
Revised: By the deﬁnition of sufﬁcient statistic we know p ( x | θθθ ) =

Original: p ( x | s ) p ( s | θθθ ) .
Revised: p ( x | s ) p ( s | θθθ ) .

Original: Then we have the Markov chain θθθ → s → X for the data generating process.
Revised: Then we have the Markov chain θθθ → s → x for the data generating process.

Original: On the other hand, since x ∼ p ( x | θθθ ) and S is a deterministic function we have the Markov chain θθθ
Revised: On the other hand, since x ∼ p ( x | θθθ ) and S is a deterministic function we have the Markov chain θθθ

Original: → X → S .
Revised: → x → s .

Original: By data processing inequality we have I ( θθθ ; s ( X )) ≤ I ( θθθ ; X ) for the ﬁrst chain and I ( θθθ ; X ) ≤ I ( θθθ ; s ( X )) for the second chain.
Revised: By data processing inequality we have I ( θθθ ; s ( x )) ≤ I ( θθθ ; x ) for the ﬁrst chain and I ( θθθ ; x ) ≤ I ( θθθ ; s ( x )) for the second chain.

Original: This implies that I ( θθθ ; X ) =
Revised: This implies that I ( θθθ ; x ) =

Original: I ( θθθ ; s ( X ))
Revised: I ( θθθ ; s ( x ))

Original: i.e s is the maximizer of I ( θθθ ; S ( X )) .
Revised: i.e s is the maximizer of I ( θθθ ; S ( x )) .

Original: For the other direction, since I ( θθθ ; s ( X ))
Revised: For the other direction, since I ( θθθ ; s ( x ))

Original: =max S : X→S
Revised: =

Original:  I ( θθθ ; S ( X )) , we have I ( θθθ ; s ( X ))
Revised: max S I ( θθθ ; S ( x )) , we have I ( θθθ ; s ( x ))

Original: =
Revised: =

Original: I ( θθθ ; X ) .
Revised: I ( θθθ ; x ) .

Original: Note that θθθ → X → s is a Markov chain, from Theorem 2.8.1 of Cover et al.
Revised: Note that θθθ → x → s is a Markov chain, from Theorem 2.8.1 of Cover et al.

Original: (2003) we can get θθθ and X is conditionally independent given s .
Revised: (2003) we can get θθθ and X is conditionally independent given s .

Original: This results in that s is sufﬁcient.
Revised: This implies s is sufﬁcient.

Original: A. P ROOF OF P ROPOSITION
Revised: A. P ROOF OF P ROPOSITION

Original: Proof .
Revised: Proof .

Original: We can rewrite theabove objective as E p ( θθθ, x ) [ (cid:107) S ( x ) − θθθ (cid:107) 22 ] = (cid:82) p ( θθθ, x ) log e (cid:107) S ( x ) − θθθ (cid:107) 

Revised: We can write the objective as E p ( θθθ, x ) [ (cid:107) S ( x ) − θθθ (cid:107) 22 ] = (cid:82) p ( θθθ, x ) log e (cid:107) S ( x ) − θθθ (cid:107)

Original: d x dθθθ .
Revised: d x dθθθ .

Original: On the other hand we have I ( θ θ θ, S ( x ))
Revised: On the other hand we have I ( θθθ ; S ( x ))

Original: = (cid:82) p ( θ θ θ, x ) log p ( S ( x ) | θ θ θ ) /p
Revised: = (cid:82) p ( θθθ, x ) log p ( S ( x ) | θθθ ) /p

Original: ( S ( x ))
Revised: ( S ( x ))

Original: d x dθ θ θ .
Revised: d x dθθθ .

Original: By comparing them it is easy to seethat they are generally not equivalent except for very rare case (e.g Gaussian).
Revised: By comparing them, we see they are generally not equivalent. Equivalence only holds in special cases (e.g. Gaussians).

Original: B M ORE EXPERIMENTAL DETAILS AND RESULTS B. D ETAILED EXPERIMENTAL SETTINGS
Revised: B M ORE E XPERIMENTAL DETAILS AND RESULTS B. D ETAILED EXPERIMENTAL SETTINGS

Original: Neural networks settings .
Revised: Neural networks settings .

Original: For the statistic network S in our method, we adopt a D -100-100-
Revised: For the statistic network S in our method (for both JSD and DC estimators), we adopt a D -100-100-

Original: d fully-connected architecture with D being the dimensionality of input data and d the dimensionality of the statistic.
Revised: d fully-connected architecture with D being the dimensionality of input data and d the dimensionality of the statistic.

Original: For the network H used to extract the representation of θθθ , we adopt a D -100-100-
Revised: For the network H used to extract the representation of θθθ , we adopt a K

Original: K fully-connected architecture with K being the dimensionality of the model parameter.
Revised: K fully-connected architecture with K being the dimensionality of the model parameters θθθ .

Original: For the critic network, we adopt a ( d + K ) -100-1 fully connected architecture with K being the dimensionality of θθθ .
Revised: For the critic network, we adopt a ( d + K ) -100-1 fully connected architecture.

Original: ReLU is adopted as the non-linearity in all networks.
Revised: ReLU is adopted as the non-linearity in all networks.

Original: For SRE, which is closely related to our method, we use a ( D + K ) -144-144-100-1 architecture.
Revised: For SRE, which is closely related to our method, we use a ( D + K ) -144-144-100-1 architecture.

Original: This architecture has a similar complexity as our networks.
Revised: This architecture has a similar complexity as our networks.

Original: All these neural networks are trained with Adam (Kingma & Ba, 2014) with a learning rate of 1 × 10 − 3 and a batch size of 100.
Revised: All these neural networks are trained with Adam (Kingma & Ba, 2014) with a learning rate of 1 × 10 − 4 and a batch size of 200.

Original: No weight decay is applied.
Revised: No weight decay is applied.

Revised: We take 20% of the data for validation, and stop training if the validation error does not improve after 100 epochs.

Revised: We take the snapshot with the best validation error as the ﬁnal result.

Original: For the neural density estimator in SNL/SNPE, which is realized by a Masked Autoregressive Flow (MAF) (Papamakarios et al., 2017), we adopt 5 autoregressive layers, each of which has two hidden layers with 50 tanh units.
Revised: For the neural density estimator in SNL/SNPE, which is realized by a Masked Autoregressive Flow (MAF) (Papamakarios et al., 2017), we adopt 5 autoregressive layers, each of which has two hidden layers with 50 tanh units.

Original: This is the same settings as in SNL.
Revised: This is the same settings as in SNL.

Original: The MAF are trained with Adam with a learning rate of 5 × 10 − 4 and a batch size of 500 and a slight weight decay ( 1 e − 5 ).
Revised: The MAF is trained with Adam with a learning rate of 5 × 10 − 4 and a batch size of 500 and a slight weight decay ( 1 × 10 − 4 ).

Original: For all networks, we take 20% of the data for validation, and stop training if the validation error does not improve after 100 epochs.
Revised: Similar to the case of MI networks, we take 20% of the data for validation, and stop training if the validation error does not improve after 100 epochs.

Original: We take the snapshot with the best validation error  as the result.
Revised: The snapshot with the best validation error is taken as the result.

Original: Sampling from the approximate posterior .
Revised: Sampling from the approximate posterior/learnt proposal .

Original: For fair comparison, we adopt simple rejection sampling for all LFI methods (ABC, SNL, SNPE, SRE) when sampling from the learnt posterior, so that each LFI algorithm only differs in the way they learn the posterior.
Revised: For fair comparison, we adopt simple rejection sampling for all LFI methods (ABC, SNL, SNPE, SRE) when sampling from the learnt posterior, so that each LFI method only differs in the way they learn the posterior.

Original: Dimensionality of the learned statistic . For the JSD/contour plots presented in the experiment part in the main text, we use d = (cid:98) D/ 5 (cid:99) , (cid:98) D/ 5 (cid:99) , (cid:98) D/
Revised: No MCMC is used.

Original: 10 (cid:99) for the Ising model problem, Gaussian copula problem and OP process problem respectively (remark that D is the dimensionality of the raw data).
Revised: Empirical version of objective functions .

Original: Training of statistic network . Remark that  the statistic network S ( · ) is trained with the following objective together with the scoring network T ( · ) :
Revised: Recall that in the JSD estimator, the statistic network S ( · ) is trained with the following objective together with the critic network T ( · ) :

Original: [Equation]
Revised: [Equation]

Original: the mini-batch approximation to this objective is:
Revised: the mini-batch approximation to this objective is:

Original: [Equation]
Revised: [Equation]

Original: where { j 1 , j 2 , ..., j n } is the j -th random permutation of the indexes { 1 , 2 , ..., n } and the pair ( θθθ i , x i ) are randomly picked from the data D
Revised: where { j 1 , j 2 , ..., j n } is the j -th random permutation of the indexes { 1 , 2 , ..., n } and the pair ( θθθ i , x i ) are randomly picked from the data D

Original: = { θθθ i , x i } N i =1 .
Revised: = { θθθ i , x i } Ni =1 .

Original: Here wesimply set m = 200 and n  = 100 .
Revised: Here we set m = 400 and n is the batch size. In the DC estimator, the statistic network is trained by the following objective: [Equation] where h ( a , b ) = (cid:107) a − b (cid:107) − E p ( b (cid:48) )

Revised: [ (cid:107) a − b (cid:48) (cid:107) ] − E p ( a (cid:48) )

Original: JSD calculation between true posterior and approximate posterior . 
Revised: [ (cid:107) a (cid:48) − b (cid:107) ] + E p ( a (cid:48) ) p ( b (cid:48) ) [ (cid:107) a (cid:48) − b (cid:48) (cid:107) ] . The mini-batch approximation to this objective is:

Revised: [Equation]

Revised: where ˜ h ( a i , b j ) =

Revised: (cid:107) a

Revised: i − b j (cid:107)− 1 n − 2 (cid:80)

Revised: [Equation]

Revised: − b j (cid:48) (cid:107) .

Original:  The calculation of the JensenShannon divergence between  true posterior P and approximate posterior Q  namely JSD ( P, Q ) = 12 KL [ P (cid:107) ( P + Q ) / 2]+ 12 KL [ Q (cid:107) ( P + Q ) / 2]  is done numerically by a Riemann sum over 30 K equally spaced grid points with K being the dimensionality of θθθ .
Revised: Here i, j, i (cid:48) , j (cid:48) are the indexes in the mini-batch. n is again the batch size. JSD calculation between true posterior and approximate posterior . The calculation of the JensenShannon divergence between the true posterior P and approximate posterior Q , namely JSD ( P, Q ) = 12 KL [ P (cid:107) ( P + Q ) / 2]+ 12 KL [ Q (cid:107) ( P + Q ) / 2] , is done numerically by a Riemann sum over 30 K equally spaced grid points with K being the dimensionality of θθθ .

Original: The region of these grid points isjointly deﬁned by the min and max values of 500 samples drawn from Pand Q .
Revised: The region of these grid points is deﬁned by the min and max values of 500 samples drawn from P .

Original: When weare only equipped with samples of the true posterior (e.g the Ising model problem), we estimate P by KDE (remark that for the Ising model the posterior is 1D).
Revised: When we only have samples from the true posterior (e.g. the Ising model), we approximate P by a mixture of Gaussian with 8 components.

Original: B. A DDITIONAL EXPERIMENTAL RESULTSComparison of different MI estimators
Revised: B. A DDITIONAL EXPERIMENTAL RESULTS

Revised: Comparison of different MI estimators .

Original: [Figure] We conduct empirical study on a 10-round Gaussian copula task with four different mutual information estimators: MINE (Belghazi et al., 2018), MINE- f (Belghazi et al., 2018), Jensen-Shannon divergence (JSD) estimators (Hjelm et al., 2018), and classical ratio trick method (Hermans et al., 2019; Thomas et al., 2016).
Revised: We compare the performances of four MI estimator for infomax statistics learning: Donsker-Varadhan (DV) estimator (Belghazi et al., 2018), JensenShannon divergence (JSD) estimator (Hjelm et al., 2018), distance correlation (DC)

Original: The results are averaged over 3 independent runs. We use the JSD between the true posterior and approximate posterior as in the main text as the metric. From Figure 4 we see the JSD estimator achieves the best performance, and as such we adopt it as the MI estimator in our method. Results under MMD metric .
Revised: Székely et al.

Original: For reference, we also conduct experiments with MMD (Gretton et al., 2012) discrepancy on Gaussian copula and OU process with different reruns as a reference.
Revised: (2014) and Wasserstein distance (WD) (Ozair et al., 2019).

Original: Our proposed methods achieve consistent performance under JSD metric (we remind the reader here that the result by JSD should be more reliable since it directly compares two analytical densities). The results are averaged over 3 independent runs [Figure] [Figure]
Revised: We highlight that the last two estimators (DC and WD) are ratio-free.

Original: On directly extracting the posterior from the MI network .

Original: As mentioned in the main text, it has been shown that the optimal critic t ( θθθ, s ) in f -divergence-based MI estimator I ( θθθ, s ) indeed corresponds to the density ratio up to a constant: p ( θθθ, s ) /p

Original: ( θθθ ) p ( s ) . In JSD-based MI estimator, which we used to learn the sufﬁcient statistics, we can extract the ratio as [Equation] with which we can recover the(sufﬁcient statistics-based) posterior estimate as:
Revised: We compare the discrepancy between the true posterior and the posterior inferred with the statistics learned by each estimator, as well as the execution time per each mini-batch.

Original: [Equation]

Original: So one might wonder how the posterior estimated in this strategy is compared to the strategy mentioned in the main text (namely re-ﬁtting a neural density estimator q ( s | θθθ ) on s ).

Original: The ﬁgures below show the results. [Equation] 0.8 θ 0 0. θ0.3 0.4 0.
Revised: The results, which are averaged over 5 independent runs, are shown in the ﬁgure and the table below.

Original: [Equation]

Original: 0.6 0.7 0.8 θ 0 0.

Original: θ0.

Original: [Equation]

Original: [Equation]

Original: 0.8 θ 0 0.

Original: [Equation] θtruth extract from T NDE reﬁt [Equation]
Revised: From the ﬁgure we see that the JSD estimator generally yields the best accuracy among the four estimators.

Original: 0.8 θ 0 0.

Original: [-θ0.1 0.2 0.3 0.4 0.5 0.
-]
Revised: [-θ0.1 0.2 0.3 0.4 0.5 0.
-]

Original: 0.7 0.8 θ 0 0. θtruth extract from T NDE reﬁt We discover that the posteriors returned by NDE reﬁt seem to be more accurate than the one directly extracted from the critic T .
Revised: In terms of execution time, the DC estimator is clearly the winner, with its execution time being only 1/15 of the other estimators.

Original: Here, the JSD in the two problems are (0.000, 0.014, 0.004) and (0.000, 0.010, 0.004) respectively (the OU process is  the same as in the main text. The wrapped Gaussian model is one where the data is generated as x ∼ N ( x , tanh ( θθθ ) , I ) . All results are obtained with 10,000 simulations). More speciﬁcally, we discover that the NDE posteriors tends to be smoother and hence closer to a realistic pdf.
Revised: However, the accuracy of the DC estimator is still comparable to the JSD estimator, especially when the number of training samples is large (e.g.

Original: We conjecture that this might due to two reasons: • In NDE reﬁt, the density is guaranteed to be a valid pdf during the learning procedure. This might be a good inductive bias that helps to learn the density better. The density extracted from the ratio on the contrary is not guaranteed to be a valid one, so it is less realistic;
Revised: According to these results, we suggest using JSD in small-scale settings (e.g. early rounds in sequential learning), and use DC in large-scale ones (e.g. later rounds in sequential learning).

Original: • It is also possible that even if the whole statistics + critic networks does not estimate the density perfectly, the statistics network itself might still do a good job in learning the sufﬁcient statistics. This opinion was also
Revised: [Figure]

Original: appreciated

Original: in

Original: previous

Original: works (see e.g Cranmer et al. Nonetheless we feel more exploration is required to draw a conclusion. We leave it to future work.
Revised: [Table]

Original: Contrastive learning v.s. MLE .
Revised: Contrastive learning v.s. MLE .

Original: In the experiment in the main text, we discover that our method does not always achieve the best performance: SNPE-B performs better  on the Gaussian copula problem than our method.
Revised: In the experiment in the main text, we discover that our method does not always achieve the best performance; it does not work better than SNPE-B on the Gaussian copula problem.

Original: Here we would like to investigate why this happens.
Revised: Here we would like to investigate why this happens.

Original: Upon a closer look, we discover that SRE, which is closely related to our method, isalso outperformed by SNPE-B on the Gaussian copula problem.
Revised: Upon a closer look, we discover that SRE, which is closely related to our method when used with the JSD estimator, is outperformed by SNPE-B on the Gaussian copula problem.

Original: Remark that both SRE and our method uses contrastive learning rather than MLE.
Revised: Remark that both SRE and our method, when used with the JSD estimator, uses contrastive learning rather than MLE.

Original: Since both of these two contrastive learning methods are outperformed by the MLE-based SNPE-B, it makes us suspect the reason is due to imperfect contrastive learning.
Revised: Since both of these two contrastive learning methods do not perform better than the MLE-based SNPE-B, it makes us suspect the reason is due to imperfect contrastive learning.

Original: To verify this, we further conduct experiments for SNPE-C, which shares the same loss function with SRE but with a different parameterization to the density ratio (SRE: fully-connected network; SNPE-C: NDE-based parameterization.
Revised: To verify this, we further conduct experiments for SNPE-C, which shares the same loss function with SRE but with a different parameterization to the density ratio (SRE: fully-connected network; SNPE-C: NDE-based parameterization.

Original: This NDE is the same as in SNL).
Revised: This NDE is the same as in SNL).

Original: The result is as follows:
Revised: The result is as follows:

Original: [Table]
Revised: [Table]

Original: Surprisingly, we ﬁnd thatlike SRE and SNL+, SNPE-C also perform less satisfactorily than SNPE-B on the Gaussian copula problem.
Revised: Surprisingly, we ﬁnd that SNPE-C also perform less satisfactorily than SNPE-B on the Gaussian copula problem.

Original: This suggests that contrastive learning might be less preferable than 
Revised: This suggests that contrastive learning might be less preferable than MLE on the

Original: MLE on this problem, which might also explain the less satisfactory performance of our method.
Revised: Gaussian copula problem, which might also explain the less satisfactory performance of our method.