Original: N EURAL A PPROXIMATE S UFFICIENT S TATISTICS FORL IKELIHOOD - FREE I NFERENCE
Revised: N EURAL A PPROXIMATE S UFFICIENT S TATISTICS FOR I MPLICIT M ODELS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We consider the fundamental problem of how to automatically construct summary statistics for likelihood-free inference where the evaluation of likelihood function is intractable but sampling / simulating data from the model is possible.
Revised: We consider the fundamental problem of how to automatically construct summary statistics for implicit generative models where the evaluation of likelihood function is intractable but sampling / simulating data from the model is possible.

Original: The idea is to frame the task of constructing sufﬁcient statistics as learning mutual information maximizing representation of the data.
Revised: The idea is to frame the task of constructing sufﬁcient statistics as learning mutual information maximizing representation of the data.

Original: This representation is computed by a deep neural network trained by a joint statistic-posterior learning strategy.
Revised: This representation is computed by a deep neural network trained by a joint statistic-posterior learning strategy.

Original: We apply our approach to both traditional approximate Bayesian computation  and recent neural likelihood methods, boosting their performance on a range of tasks.
Revised: We apply our approach to both traditional approximate Bayesian computation (ABC) and recent neural likelihood approaches, boosting their performance on a range of tasks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Many data generating processes can be well-described by a parametric statistical model that can be easily simulated forward but does not possess an analytical likelihood function.
Revised: Many data generating processes can be well-described by a parametric statistical model that can be easily simulated forward but does not possess an analytical likelihood function.

Original: These models are called implicit generative models (Diggle & Gratton, 1984) or simulator-based models (Lintusaari et al., 2017) and are widely used in science and engineering domains, including physics (Sjöstrand et al., 2008), genetics (Järvenpää et al., 2018), computer graphics (Mansinghka et al., 2013), ﬁnance (Bansal & Yaron, 2004), cosmology (Weyant et al., 2013), ecology (Wood, 2010) and epidemiology (Chinazzi et al., 2020).
Revised: These models are called implicit generative models (Diggle & Gratton, 1984) or simulator-based models (Lintusaari et al., 2017) and are widely used in science and engineering domains, including physics (Sjöstrand et al., 2008), genetics (Järvenpää et al., 2018), computer graphics (Mansinghka et al., 2013), ﬁnance (Bansal & Yaron, 2004), cosmology (Weyant et al., 2013), ecology (Wood, 2010) and epidemiology (Chinazzi et al., 2020).

Original: For example, the number of infected/healthy people in an outbreak could be well modelled by stochastic differential equations (SDE) simulated by Euler-Maruyama discretization but the likelihood function of a SDE is generally non-analytical.
Revised: For example, the number of infected/healthy people in an outbreak could be well modelled by stochastic differential equations (SDE) simulated by Euler-Maruyama discretization but the likelihood function of a SDE is generally non-analytical.

Original: Directly inferring the parameters of these implicit models is often difﬁcult or even impossible.
Revised: Directly inferring the parameters of these implicit models is often difﬁcult or even impossible.

Original: The techniques coined as likelihood-free inference open  a door for performing  inference in such circumstances.
Revised: The techniques coined as likelihood-free inference open us a door for performing Bayesian inference in such circumstances.

Original: Likelihood-free inference needs to evaluate neither the likelihood function nor its derivatives.
Revised: Likelihood-free inference needs to evaluate neither the likelihood function nor its derivatives.

Original: Rather, it only requires the ability to sample ( i.e. simulate) data from the model.
Revised: Rather, it only requires the ability to sample ( i.e. simulate) data from the model.

Original: Early approaches in approximate Bayesian computation (ABC) perform likelihood-free inference by repeatedly simulating data from the model, and pick a small subset of the simulated data close to the observed data to build the posterior (Pritchard et al., 1999;
Revised: Early approaches in approximate Bayesian computation (ABC) perform likelihood-free inference by repeatedly simulating data from the model, and pick a small subset of the simulated data close to the observed data to build the posterior (Pritchard et al., 1999;

Original: Marjoram et al., 2003; Beaumont et al., 2009; Sisson et al., 2007).
Revised: Marjoram et al., 2003; Beaumont et al., 2009; Sisson et al., 2007).

Original: Recent advances make use of ﬂexible neural density estimators to approximate either the intractable likelihood (Papamakarios et al., 2019) or  the posterior (Papamakarios & Murray, 2016; Lueckmann et al., 2017; Greenberg et al., 2019).
Revised: Recent advances make use of ﬂexible neural density estimators to approximate either the intractable likelihood (Papamakarios et al., 2019) or directly the posterior (Papamakarios & Murray, 2016; Lueckmann et al., 2017; Greenberg et al., 2019).

Original: Despite the algorithmic differences, a shared ingredient in likelihood-free inference methods is the choice of summary statistics.
Revised: Despite the algorithmic differences, a shared ingredient in likelihood-free inference methods is the choice of summary statistics.

Original: Well-chosen summary statistics have been proven crucial for the performance of likelihood-free inference methods (Blum et al., 2013; Fearnhead & Prangle, 2012;
Revised: Well-chosen summary statistics have been proven crucial for the performance of likelihood-free inference methods (Blum et al., 2013; Fearnhead & Prangle, 2012;

Original: Sisson et al., 2018).
Revised: Sisson et al., 2018).

Original: Unfortunately, in practice it is often difﬁcult to determine low-dimensional and informative summary statistic without domain knowledge from experts.
Revised: Unfortunately, in practice it is often difﬁcult to determine low-dimensional and informative summary statistic without domain knowledge from experts.

Original: In this work, we propose a novel deep neural network-based approach for automatic construction of summary statistics.
Revised: In this work, we propose a novel deep neural network-based approach for automatic construction of summary statistics.

Original: Neural networks have been previously applied to learning summary statistics for likelihood-free inference (Jiang et al., 2017; Dinev & Gutmann, 2018; Alsing et al., 2018; Brehmer et al., 2020).
Revised: Neural networks have been previously applied to learning summary statistics for likelihood-free inference (Jiang et al., 2017; Dinev & Gutmann, 2018; Alsing et al., 2018; Brehmer et al., 2020).

Original: Our approach is unique in that our learned statistics directly target at global sufﬁciency .
Revised: Our approach is unique in that our learned statistics directly target at global sufﬁciency .

Original: The main idea is to exploit the link between statistical sufﬁciency and information theory, and to formulate the task of learning sufﬁcient statistic as the task of learning information-maximizing representation of data.
Revised: The main idea is to exploit the link between statistical sufﬁciency and information theory, and to formulate the task of learning sufﬁcient statistic as the task of learning information-maximizing representation of data.

Original: We achieve this with recent neural mutual information estimators (Belghazi et al., 2018; Hjelm et al., 2018).
Revised: We achieve this with recent neural mutual information estimators (Belghazi et al., 2018; Hjelm et al., 2018).

Original: Importantly, such neural sufﬁcient statistic could be learned jointly with the posterior, resulting in fast learning where the two could reﬁne each other iteratively.
Revised: Importantly, such neural sufﬁcient statistic could be learned jointly with the posterior, resulting in fast learning where the two could reﬁne each other iteratively.

Original: To sum up, our main contributions are:
Revised: To sum up, our main contributions are:

Original: • We propose a new neural approach to automatically extract compact, near-sufﬁcient statistics from raw data.
Revised: • We propose a new neural approach to automatically extract compact, near-sufﬁcient statistics from raw data.

Original: The approach removes the need for carefully handcrafted design of summary statistics; • With the proposed statistics, we develop two new likelihood-free inference methods namely SMCABC+ and SNL+.
Revised: The approach removes the need for carefully handcrafted design of summary statistics; • With the proposed statistics, we develop two new likelihood-free inference methods namely SMCABC+ and SNL+.

Original: Experiments on tasks with various types of data demonstrate their effectiveness.
Revised: Experiments on tasks with various types of data demonstrate their effectiveness.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: Likelihood-free inference .
Revised: Likelihood-free inference .

Original: LFI considers the task of Bayesian inference where the likelihood function of the model is intractable but simulating (sampling) data from the model is possible:
Revised: LFI considers the task of Bayesian inference where the likelihood function of the model is intractable but simulating (sampling) data from the model is possible:

Original: [Equation]
Revised: [Equation]

Original: where x o is the observed data, π ( θθθ ) is the prior over the model parameters θθθ , p ( x o | θθθ ) is the (possibly) non-analytical likelihood function and π ( θθθ | x o ) is the posterior over θθθ .
Revised: where x o is the observed data, π ( θθθ ) is the prior over the model parameters θθθ , p ( x o | θθθ ) is the (possibly) non-analytical likelihood function and π ( θθθ | x o ) is the posterior over θθθ .

Original: We assume while we do not have access to the exact likelihood, we could still sample  data from the model with a simulator: x ∼ p ( x | θ θ θ ) .
Revised: We assume while we do not have access to the exact likelihood, we could still sample (simulate) data from the model with a simulator: x ∼ p ( x | θθθ ) .

Original: The task is then to infer π ( θ θ θ | x o ) given x o and the sampled data: D =
Revised: The task is then to infer π ( θθθ | x o ) given x o and the sampled data: D =

Original: { θ θ θ i , x i } n i =1 where θθθ i ∼ p ( θθθ ) , x i ∼ p ( x | θθθ i ) .
Revised: { θ θ θ i , x i } ni =1 where θ θ θ i ∼ p ( θ θ θ ) , x i ∼ p ( x | θ θ θ i ) .

Original: Note that p ( θθθ ) is not necessarily the prior π ( θθθ ) .
Revised: Note that p ( θ θ θ ) is not necessarily the prior π ( θ θ θ ) .

Original: Curse of dimensionality .
Revised: Curse of dimensionality .

Original: Different likelihood-free inference algorithms might learn π ( θθθ | x o ) in different ways, nevertheless most of existing methods suffer from the curse of dimensionality.
Revised: Different likelihood-free inference algorithms might learn π ( θθθ | x o ) in different ways, nevertheless most of existing methods suffer from the curse of dimensionality.

Original: For example, traditional ABC methods use a small subset of D closest to x o under some metric to build the posterior (Pritchard et al., 1999; Marjoram et al., 2003;
Revised: For example, traditional ABC methods use a small subset of D closest to x o under some metric to build the posterior (Pritchard et al., 1999; Marjoram et al., 2003;

Original: Beaumont et al., 2009; Sisson et al., 2007), however in high-dimensional space measuring the distance sensibly is notoriously hard (Sorzano et al., 2014; Xie et al., 2017).
Revised: Beaumont et al., 2009; Sisson et al., 2007), however in high-dimensional space measuring the distance sensibly is notoriously hard (Sorzano et al., 2014; Xie et al., 2017).

Original: On the other hand, recent advances (Papamakarios et al., 2019; Lueckmann et al., 2017; Papamakarios & Murray, 2016) utilize neural density estimators (NDE) to model the intractable likelihood or the posterior.
Revised: On the other hand, recent advances (Papamakarios et al., 2019; Lueckmann et al., 2017; Papamakarios & Murray, 2016; Greenberg et al., 2019) utilize neural density estimators (NDE) to model the intractable likelihood or the posterior.

Original: Unfortunately, modeling high-dimensional distributions with NDE accurately is also known to be  difﬁcult (Oord et al., 2016; 
Revised: Unfortunately, modeling high-dimensional distributions with NDE accurately is also known to be very difﬁcult (Oord et al., 2016; Rippel &

Original: Rippel & Adams, 2013; Oord et al., 2017), especially when the available training (simulated) data is scarce.
Revised: Adams, 2013; Oord et al., 2017), especially when the available training (simulated) data is scarce.

Original: Our interest here is not to design a new inference algorithm, but to ﬁnd a low-dimensionaland a near-sufﬁcient statistic s = s ( x ) that satisﬁes:
Revised: Our interest here is not to design a new inference algorithm, but to ﬁnd a low-dimensional statistic s = s ( x ) that is (Bayesian) sufﬁcient:

Original: [Equation]
Revised: [Equation]

Original: where s : X → S is a deterministic function also learned from D .
Revised: where s : X → S is a deterministic function also learned from D .

Original: We conjecture that the learning of s ( · ) might be an easier task than direct density estimation.
Revised: We conjecture that the learning of s ( · ) might be an easier task than direct density estimation.

Original: The resultant statistic s could then be applied to a wide range of likelihood-free inference algorithms as we  elaborate in Section 3.2.
Revised: The resultant statistic s could then be applied to a wide range of likelihood-free inference algorithms as we will elaborate in Section 3.2.

Original:  M ETHODOLOGY 3. N EURAL SUFFICIENT STATISTICS
Revised:  M ETHODOLOGY 3. N EURAL SUFFICIENT STATISTICS

Original: Our new deep neural network-based approach for automatically construction of (near-)sufﬁcient statistics is based on the infomax principle, as illustrated by the following proposition :
Revised: Our new deep neural network-based approach for automatic construction of (near-)sufﬁcient statistics is based on the infomax principle, as illustrated by the following proposition :

Original: Proposition 1.
Revised: Proposition 1.

Original: Let θθθ ∼ p ( θθθ ) , x ∼ p ( x | θθθ ) , and s : X → S be a deterministic function.
Revised: Let θθθ ∼ p ( θθθ ) , x ∼ p ( x | θθθ ) , and s : X → S be a deterministic function.

Original: Then s = s ( x ) is a sufﬁcient statistic for p ( x | θθθ )
Revised: Then s = s ( x ) is a sufﬁcient statistic for p ( x | θθθ )

Original: if and only if
Revised: if and only if

Original: [Equation]
Revised: [Equation]

Original: where S is deterministic mapping and I ( · , · ) is the mutual information between random variables.
Revised: where S is deterministic mapping and I ( · , · ) is the mutual information between random variables.

Original: Proof .
Revised: Proof .

Original: We defer the complete proof to the Appendix.
Revised: We defer the complete proof to the Appendix.

Original: This proposition is a variant of Theorem 8 in (Shamir et al., 2008) with an adaption to likelihood-free inference scenario.
Revised: This proposition is a variant of Theorem 8 in (Shamir et al., 2008) with an adaption to likelihood-free inference scenario.

Original: This important result suggests that we could ﬁnd the sufﬁcient statistic s ( x ) for a likelihood function p ( x | θθθ ) by maximizing the mutual information (MI) I ( θθθ ; S ) =
Revised: This important result suggests that we could ﬁnd the sufﬁcient statistic s ( x ) for a likelihood function p ( x | θθθ ) by maximizing the mutual information (MI) I ( θθθ ; S ) =

Original: KL [ p ( θθθ, S ) (cid:107) p ( θθθ ) p
Revised: KL [ p ( θθθ, S ) (cid:107) p ( θθθ ) p

Original: ( S )] between θθθ and S .
Revised: ( S )] between θθθ and S .

Original: Moreover, as our interest is in maximizing MI rather than knowing its precise value, we can maximize a non-KL surrogate, which has been proven to yield better performance (Hjelm et al., 2018;
Revised: Moreover, as our interest is in maximizing MI rather than knowing its precise value, we can maximize a non-KL surrogate, which has been proven to yield better performance (Hjelm et al., 2018;

Original: Ozair et al., 2019; Wen et al., 2020).
Revised: Ozair et al., 2019; Wen et al., 2020).

Original: To this end, we utilize the Jensen-Shannon divergence (JSD) surrogate (Hjelm et al., 2018), a distribution-free, non-KL MI estimator.
Revised: To this end, we utilize the Jensen-Shannon divergence (JSD) surrogate (Hjelm et al., 2018), a distribution-free, non-KL MI estimator.

Revised: This estimator was shown to be much less sensitive to sample size (Hjelm et al., 2018).

Original: It estimates MI as:
Revised: More speciﬁcally, it estimates MI as:

Original: [Equation]
Revised: [Equation]

Original: where sp( t ) = log(1 + exp( t )) is the softplus function.
Revised: where sp( t ) = log(1 + exp( t )) is the softplus function.

Original: With this estimator, we set up the following objective for learning the sufﬁcient statistic, which simultaneously estimates and maximizes the MI:
Revised: With this estimator, we set up the following objective for learning the sufﬁcient statistics, which simultaneously estimates and maximizes the MI:

Original: [Equation]
Revised: [Equation]

Original: where the two deterministic mappings S and T are parameterized by two neural networks.
Revised: where the two deterministic mappings S and T are parameterized by two neural networks.

Original: Note that we have used the law of the unconscious statistician from equation 3 to equation 4.

Original: With considerable training samples and powerful neural networks S and T , we could obtain nearsufﬁcient statistic with s = arg max S max T L ( S, T ) .
Revised: With considerable training samples and powerful neural networks S and T , we could obtain nearsufﬁcient statistic with s = arg max S max T L ( S, T ) .

Original: The statistic s of data x is then given bys =
Revised: The statistic s of data x is then given by

Original: [Equation]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: In the above construction, we have not speciﬁed the form of the networks S and T .
Revised: In the above construction, we have not speciﬁed the form of the networks S and T .

Original: For T , we choose it to be a split architecture:
Revised: For T , we choose it to be a split architecture:

Original: [Equation]
Revised: [Equation]

Original: where T (cid:48) ( · , · ) , H ( · ) are both MLPs.
Revised: where T (cid:48) ( · , · ) , H ( · ) are both MLPs.

Original: For S , any prior knowledge about the data x could in principle be incorporated into its design.
Revised: For S , any prior knowledge about the data x could in principle be incorporated into its design.

Original: For example, for sequential data(e.g data simulated from a stochastic dynamical system) we can realize S as a LSTM network (Gers et al., 1999), and for exchangeable data(e.g gene data) we can realize S as a exchangeable neural network (Chan et al., 2018).
Revised: For example, for sequential data we can realize S as a LSTM network (Gers et al., 1999), and for exchangeable data we can realize S as a exchangeable neural network (Chan et al., 2018).

Original: Here we simply adopt a fully-connected architecture for S , and leave the problem-speciﬁc design of S as future works.
Revised: Here we simply adopt a fully-connected architecture for S , and leave the problem-speciﬁc design of S as future work.

Original: Therefore we separately learn representations for x and θθθ before processing them together.
Revised: Therefore we separately learn representations for x and θθθ before processing them together.

Original: This could be seen as that we incorporate the inductive bias into the design of the networks that x and θθθ should not interact with each other directly, based on their true relationship (for example, consider the exponential family distribution where L ( θθθ ; x ) ∝ exp ( H ( θθθ ) (cid:62) S ( x )) ).
Revised: This could be seen as that we incorporate the inductive bias into the design of the networks that x and θθθ should not interact with each other directly, based on their true relationship, (for example, consider the exponential family: L ( θθθ ; x ) ∝ exp ( H ( θθθ ) (cid:62) S ( x )) ).

Original: We are left with the problem of how to select d , the dimensionality of the sufﬁcient statistics.
Revised: We are left with the problem of how to select d , the dimensionality of the sufﬁcient statistics.

Revised: The

Revised: Pitman-Koopman-Darmois theorem tells us that sufﬁcient statistics with ﬁxed dimensionality only exists for exponential family, so one might better determine d in a data-driven way.

Original: In principle, we wish d to be as small as possible while preserving enough information about θθθ , namely
Revised: To do this, we can in principle set d to be as low as possible while preserving enough information about θθθ :

Original: [Equation]
Revised: [Equation]

Original: where s d (cid:48) is the statisticfunction withoutput dimensionality d (cid:48) , d ∗ is the dimensionality that achieves the highest MIvalue on the validation set and λ  quantiﬁes how much information we are to preserve (e.g 90% ), respectively.
Revised: where s d (cid:48) is the statistic with dimensionality d (cid:48) , d ∗ is the dimensionality that achieves the highest MI and λ ∈ (0 , 1) quantiﬁes how much information we are to preserve in the learnt statistics.

Original: Due to space limit we do not pursue this idea further in this paper and leave the veriﬁcationof this strategy to future work.
Revised: Due to space limit we do not pursue this idea further in this paper and leave its veriﬁcation to future work.

Original: Furthermore, we have the following proposition comparing our method to the existing posteriormean-as-statistic approaches (Fearnhead & Prangle, 2012; Jiang et al., 2017).
Revised: Furthermore, we have the following proposition comparing our method to the existing posteriormean-as-statistic approaches (Fearnhead & Prangle, 2012; Jiang et al., 2017).

Original: Proposition 2.
Revised: Proposition 2.

Original: Let θθθ ∼ p ( θθθ ) and x ∼ p ( x | θθθ ) .
Revised: Let θθθ ∼ p ( θθθ ) and x ∼ p ( x | θθθ ) .

Original: Let s ( · ) be the maximizer of the following objective:
Revised: Let s ( · ) be the maximizer of the following objective:

Original: [Equation]
Revised: [Equation]

Original: then s = s ( x ) is generally not a maximizer of I ( S ( x ) ,θθθ ) and hence it is not a sufﬁcient statistic.
Revised: then s = s ( x ) is generally not a maximizer of I ( S ( x ) ,θθθ ) and hence it is not a sufﬁcient statistic.

Original: Proof .
Revised: Proof .

Original: We defer the proof to the Appendix.
Revised: We defer the proof to the Appendix.

Original: This proposition tells us that unlike our method, the existing posterior-mean-as-statistic approaches widely used in likelihood-free inference community indeed loose information about the posterior, and it is only optimal for predicting the posterior mean (Fearnhead & Prangle, 2012; Jiang et al., 2017). 
Revised: This proposition tells us that unlike our method, the existing posterior-mean-as-statistic approaches widely used in likelihood-free inference community indeed loose information about the posterior, and it is only optimal for predicting the posterior mean (Fearnhead & Prangle, 2012; Jiang et al., 2017). Nonetheless which method to use depends on the task (e.g full posterior estimate v.s point estimate).

Original: 3. D YNAMIC STATISTIC - POSTERIOR LEARNING
Revised: 3. D YNAMIC STATISTIC - POSTERIOR LEARNING

Original: The above neural sufﬁcient statistic could, in principle, be learned via a pilot run before the inference starts, as typically done in the work by Drovandi et al.
Revised: The above neural sufﬁcient statistic could, in principle, be learned via a pilot run before the inference starts, as typically done in the work by Drovandi et al.

Original: (2011); Fearnhead & Prangle (2012); Jiang et al.
Revised: (2011); Fearnhead & Prangle (2012); Jiang et al.

Original: Such a strategy requires extra simulation cost, and the learned statistic is kept ﬁxed duringthe inference.
Revised: Such a strategy requires extra simulation cost, and the learned statistic is kept ﬁxed during inference.

Original: We propose a new learning strategy below to overcome these limitations.
Revised: We propose a dynamic learning strategy below to overcome these limitations.

Original: Our idea is to jointly learn the statistic and the posterior in multiple rounds.
Revised: Our idea is to jointly learn the statistic and the posterior in multiple rounds .

Original: More concretely, at round r , we use the current statistic s ( · ) to build the r -th estimate to the posterior: q r ( θθθ | s o )
Revised: More concretely, at round r , we use the current statistic s ( · ) to build the r -th estimate to the posterior: q r ( θθθ | s o )

Original: ≈
Revised: ≈

Original: π ( θθθ | x o ) , 
Revised: π ( θθθ | x o ) , sort ﬁt j

Original: Input: prior π ( θθθ ) , observed data x o Output: estimated posterior ˆ π ( θθθ | x o ) Initialization: D = ∅ , p 1 ( θθθ ) = π ( θθθ ) for j in 1 to r do repeat sample
Revised: D ← D ∪ { θθθ x ni ﬁt statistic ﬁt

Original: θθθ ( i ) ∼ p j ( θθθ  ) ; simulate x ( i ) ∼ p ( x | θθθi ) 
Revised: ; q j ( θθθ | s o ) ∝ π ( θθθ ) · q ( s o | θθθ ) ; p j +1 (

Original: ; until n samples D ← D ∪ { θθθ i , x i } ni = ﬁt statistic net s ( · ) with D by equation 4 ; sort D according to (cid:107) s ( x i ) − s ( x o ) (cid:107) ; ﬁt p ( θθθ | s o ) with the top m θθθ s in D ; q j ( θθθ | s o ) ∝ π ( θθθ ) / (cid:80) j p j ( θθθ ) · p ( θθθ
Revised: [Equation]

Original: | s o ) ; p j +1 ( θθθ ) ← q j (θθθ | s o ) ; end for return θθθ x o ) r θθθ s o ) Input: prior π ( θθθ ) , observed data x o Output: estimated posterior ˆ π ( θθθ | x o )
Revised: ← q j (

Original: Initialization: D = ∅ , p 1 ( θθθ ) = π ( θθθ ) for j in 1 to r do repeat sample θθθ ( i ) ∼ p j ( θθθ ) ; simulate x ( i ) ∼ p
Revised: [Equation]

Original: ( x | θθθ i ) ; until n samples
Revised: | s o ) ; end for return θθθ | x o )

Original: D ← D ∪ { θθθ i , x i } ni =ﬁt statistic net s ( · ) with D by equation 4; convert D with the learned s ( · ) ; ﬁt q ( s | θθθ ) with converted D by equation 10; q j ( θθθ | s o )
Revised: =

Original: ∝ π ( θθθ ) · q  (  s o| θθθ ) ; p j +1 ( θθθ ) ← q j ( θθθ | s o ) ; end for return ˆ π ( θ θ θ | x o [Equation] = q r ( θ
Revised: q r ( θθθ | s o ) whereas at round r +1, this estimate is used as the new proposal distribution to simulate data: p r +1 ( θθθ ) ← q r ( θθθ | s o ) ,θθθ i ∼ p

Original: [Equation]

Original: θ

Original: [Equation]

Original: s
Revised: r

Original: [Equation]{++
+}
Revised: {++
+}

Original: ) whereas at round r +1, this estimate is used as the new proposal distribution to simulate data: p r +1 ( θθθ ) ← q r ( θθθ | s o ) ,θθθ i ∼ p r +1 ( θθθ ) , x i ∼ p ( x | θθθ i ) .
Revised: ( θθθ ) , x i ∼ p ( x | θθθ i ) .

Original: We then re-learn s ( · ) and q ( · ) with all the data up to the new round.
Revised: We then re-learn s ( · ) and q ( · ) with all the data up to the new round.

Original: In this process, s ( · ) and q ( · ) reﬁne each other: a good s ( · ) helps to learn q ( · ) more accurately, whereas an improved q ( · ) as a better proposal in turn helps to learn s ( · ) more efﬁciently.
Revised: In this process, s ( · ) and q ( · ) reﬁne each other: a good s ( · ) helps to learn q ( · ) more accurately, whereas an improved q ( · ) as a better proposal in turn helps to learn s ( · ) more efﬁciently.

Original: In practice, any likelihood-free inference algorithm that learns the posterior sequentially naturally ﬁts well within the above joint statistic-posterior learning framework.
Revised: In practice, any likelihood-free inference algorithm that learns the posterior sequentially naturally ﬁts well within the above joint statistic-posterior learning framework.

Original: Here we study two such instances:
Revised: Here we study two such instances:

Original: Sequential Monte Carlo ABC (SMC-ABC) (Beaumont et al., 2009).
Revised: Sequential Monte Carlo ABC (SMC-ABC) (Beaumont et al., 2009).

Original: This classical algorithm learns the posterior in a non-parametric way within multiple rounds.
Revised: This classical algorithm learns the posterior in a non-parametric way within multiple rounds.

Original: Here, we consider a variant of it to better make use of the above neural sufﬁcient statistic, and to re-use all previous simulated data.
Revised: Here, we consider a variant of it to better make use of the above neural sufﬁcient statistic, and to re-use all previous simulated data.

Original: The new SMC-ABC algorithm estimates the posterior q r ( θθθ | s o ) at the r -th round as follows.
Revised: The new SMC-ABC algorithm estimates the posterior q r ( θθθ | s o ) at the r -th round as follows.

Original: We ﬁrst sort data in D
Revised: We ﬁrst sort data in D

Original: = { x i ,θθθ i } nri =1 according to the distances (cid:107) s ( x i ) − s ( x o ) (cid:107) .
Revised: = { x i ,θθθ i } nri =1 according to the distances (cid:107) s ( x i ) − s ( x o ) (cid:107) .

Original: We then pick the top- m θθθ s whose corresponding distances are the smallest.
Revised: We then pick the top- m θθθ s whose corresponding distances are the smallest.

Original: The picked θθθ s then followthe distribution θθθ ∼ p ( θθθ | s o ) as below:
Revised: The picked θθθ s then follow θθθ ∼ p ( θθθ | s o ) as below:

Original: [Equation]
Revised: [Equation]

Original: where the threshold (cid:15) is implicitly deﬁned by the ratio mnr (which automatically goes to zero as r → ∞ ).
Revised: where the threshold (cid:15) is implicitly deﬁned by the ratio mnr (which automatically goes to zero as r → ∞ ).

Original: We then ﬁt p ( θθθ | s o ) with the collected θθθ s by a ﬂexible parametric model (e.g. a Gaussian copula), with which we can obtain the r -th estimate to the posterior by importance (re-)weighting:
Revised: We then ﬁt p ( θθθ | s o ) with the collected θθθ s by a ﬂexible parametric model (e.g. a Gaussian copula), with which we can obtain the r -th estimate to the posterior by importance (re-)weighting:

Original: [Equation]
Revised: [Equation]

Original: The whole procedure of the new inference algorithm, SMC-ABC+, is summarized in Algorithm 1.
Revised: The whole procedure of the new inference algorithm, SMC-ABC+, is summarized in Algorithm 1.

Original: Sequential Neural Likelihood (SNL) (Papamakarios et al., 2019).
Revised: Sequential Neural Likelihood (SNL) (Papamakarios et al., 2019).

Original: This recent algorithm learns the posterior in a parametric way, also in multiple rounds.
Revised: This recent algorithm learns the posterior in a parametric way, also in multiple rounds.

Original: The original SNL method approximates the likelihood function p ( x | θθθ ) by a conditional neural density estimator q ( x | θθθ ) , which could be difﬁcult to learn if the dimensionality of x is high.
Revised: The original SNL method approximates the likelihood function p ( x | θθθ ) by a conditional neural density estimator q ( x | θθθ ) , which could be difﬁcult to learn if the dimensionality of x is high.

Original: Here, we alleviate such difﬁculty with our neural statistic.
Revised: Here, we alleviate such difﬁculty with our neural statistic.

Original: The new SNL algorithm estimates the posterior q r ( θθθ | s o ) at the r -th round as follows.
Revised: The new SNL algorithm estimates the posterior q r ( θθθ | s o ) at the r -th round as follows.

Original: At round r , where we have nr simulated data D = { θθθ i , x i } nri =1 , we ﬁt a neural density estimator q ( s | θθθ ) as:
Revised: At round r , where we have nr simulated data D = { θθθ i , x i } nri =1 , we ﬁt a neural density estimator q ( s | θθθ ) as:

Original: [Equation]
Revised: [Equation]

Original: where s ( · ) is the current statistic network.
Revised: where s ( · ) is the current statistic network.

Original: With nr being moderately large, this would yield us q ( s | θθθ )
Revised: With nr being moderately large, this would yield us q ( s | θθθ )

Original: ≈ p
Revised: ≈ p

Original: ( s | θθθ ) .
Revised: ( s | θθθ ) .

Original: We then obtain the r -th estimate of the posterior by Bayes rule:
Revised: We then obtain the r -th estimate of the posterior by Bayes rule:

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The whole procedure of this new SNL algorithm, denoted as SNL+, is summarized in Algorithm 2.
Revised: The whole procedure of this new SNL algorithm, denoted as SNL+, is summarized in Algorithm 2.

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: Approximate Bayesian computation .
Revised: Approximate Bayesian computation .

Original: ABC denotes techniques for likelihood-free inference which work by repeatedly simulating data from the model and picking those data similar to the observed data to estimate the posterior (Sisson et al., 2018).
Revised: ABC denotes techniques for likelihood-free inference which work by repeatedly simulating data from the model and picking those data similar to the observed data to estimate the posterior (Sisson et al., 2018).

Original: Naive ABC performs simulation with the prior, whereas advanced variants like MCMC-ABC (Marjoram et al., 2003; Meeds et al., 2015) and SMC-ABC (Beaumont et al., 2009; Sisson et al., 2007) perform simulation with informed proposals.
Revised: Naive ABC performs simulation with the prior, whereas advanced variants like MCMC-ABC (Marjoram et al., 2003; Meeds et al., 2015) and SMC-ABC (Beaumont et al., 2009; Sisson et al., 2007) perform simulation with informed proposals.

Original: To measure the similarity to the observed data sensibly, it is often wise to use low-dimensional summary statistic rather than the raw data in ABC.
Revised: To measure the similarity to the observed data sensibly, it is often wise to use low-dimensional summary statistic rather than the raw data in ABC.

Original: Here we develop a way to learn compact sufﬁcient statistic for ABC.
Revised: Here we develop a way to learn compact sufﬁcient statistic for ABC.

Original: Neural density estimator-based inference .
Revised: Neural density estimator-based inference .

Original: Apart from ABC, a recent line of research uses a conditional neural density estimator to (sequentially) learn the intractable likelihood (e.g SNL Papamakarios et al.
Revised: Apart from ABC, a recent line of research uses a conditional neural density estimator to (sequentially) learn the intractable likelihood (e.g SNL Papamakarios et al.

Original: ; Lueckmann et al.
Revised: ; Lueckmann et al.

Original: or directly the posterior (e.g SNPE Papamakarios &
Revised: or directly the posterior (e.g SNPE Papamakarios &

Original: Murray (2016); Lueckmann et al. (2017); Greenberg et al.
Revised: Murray (2016); Lueckmann et al. (2017); Greenberg et al.

Original: Likelihood-targeting approaches has the advantage that it could readily make use of any proposal distribution in sequential learning, but relies on low-dimensional, well-chosen summary statistic.
Revised: Likelihood-targeting approaches has the advantage that they could readily make use of any proposal distribution in sequential learning, but rely on low-dimensional, well-chosen summary statistic.

Original: Posterior-targeting methods on the contrary need no design of summary statistic, but they require non-trivial efforts to facilitate sequential learning.
Revised: Posterior-targeting methods on the contrary need no design of summary statistic, but they require non-trivial efforts to facilitate sequential learning.

Original: Our approach (e.g SNL+) could be seen as taking the advantages from both worlds.
Revised: Our approach (e.g SNL+) could be seen as taking the advantages from both worlds.

Original: Automatic construction of summary statistics .
Revised: Automatic construction of summary statistics .

Original: A set of works have been proposed to automatically construct low-dimensional summary statistics.
Revised: A set of works have been proposed to automatically construct low-dimensional summary statistics.

Original: Two lines of them are most related to our approach.
Revised: Two lines of them are most related to our approach.

Original: The ﬁrst line (Fearnhead & Prangle, 2012) and its variants (Jiang et al., 2017; Chan et al., 2018;
Revised: The ﬁrst line (Fearnhead & Prangle, 2012) and its variants (Jiang et al., 2017; Chan et al., 2018;

Original: Wiqvist et al., 2019; Dinev & Gutmann, 2018) train a neural network to predict the posterior mean and use this prediction as the summary statistic.
Revised: Wiqvist et al., 2019; Dinev & Gutmann, 2018) train a neural network to predict the posterior mean and use this prediction as the summary statistic.

Original: These mean-as-statistic approaches, as analyzed previously in Proposition 2, indeed do not guarantee sufﬁciency.
Revised: These mean-as-statistic approaches, as analyzed previously in Proposition 2, indeed do not guarantee sufﬁciency.

Original: Rather than taking the predicted mean, the works (Alsing et al., 2018; Brehmer et al., 2020) take the score function ∇ θθθ log p ( x | θθθ ) | θθθ = θθθ ∗ around some ﬁducial parameter θθθ ∗ as the summary statistic.
Revised: Rather than taking the predicted mean, the works (Alsing et al., 2018; Brehmer et al., 2020) take the score function ∇ θθθ log p ( x | θθθ ) | θθθ = θθθ ∗ around some ﬁducial parameter θθθ ∗ as the summary statistic.

Original: However, these score-as-statistic approaches are only locally sufﬁcient around θθθ ∗ , and it requires to ﬁrst identify θθθ ∗ which could itself be difﬁcult.
Revised: However, these score-as-statistic approaches are only locally sufﬁcient around θθθ ∗ , and it requires to ﬁrst identify θθθ ∗ which could itself be difﬁcult.

Original: Our approach differs from all these methods as it is globally sufﬁcient for all θθθ .
Revised: Our approach differs from all these methods as it is globally sufﬁcient for all θθθ .

Original: Mutual information and ratio estimation .
Revised: Mutual information and ratio estimation .

Original: It has been shown in the literature that many variational
Revised: It has been shown in the literature that many variational

Original: MI estimators I ( X ; Y ) also estimate the ratio p
Revised: MI estimators I ( X ; Y ) also estimate the ratio p

Original: ( X, Y ) /p
Revised: ( X, Y ) /p

Original: ( X ) p ( Y ) up to a constant (Nowozin et al., 2016; Nguyen et al., 2010).
Revised: ( X ) p ( Y ) up to a constant (Nowozin et al., 2016; Nguyen et al., 2010).

Original: Therefore our MI-based statistic learning method is closely related to ratio estimation approaches like (Cranmer et al., 2015;
Revised: Therefore our MI-based statistic learning method is closely related to ratio estimation approaches like (Cranmer et al., 2015;

Original: Thomas et al., 2016; Hermans et al., 2019).
Revised: Thomas et al., 2016; Hermans et al., 2019).

Original: The differences are 1) we estimate the ratio in the low-dimensional space rather than in the original space, which is typically much easier as well as more sensible; 2) we decouple the task of statistic learning from the task of density estimation, which grants us the privilege to use any infomax representation learning strategy even if they do not return the ratio, e.g Ozair et al.
Revised: The differences are 1) we estimate the ratio in the low-dimensional space rather than in the original space, based on a novel sufﬁcient statistics prospective; 2) we decouple the task of statistic learning from the task of density estimation, which grants us the privilege to use any infomax representation learning techniques even if they do not return the ratio, e.g Ozair et al.

Original: Wen et al.
Revised: Wen et al.

Original:  E XPERIMENTS 5. S ETUP
Revised:  E XPERIMENTS 5. S ETUP

Original: Baselines .
Revised: Baselines .

Original: We apply the proposed statistic to two aforementioned likelihood-free inference methods: (i)
Revised: We apply the proposed statistic to two aforementioned likelihood-free inference methods: (i)

Original: SMC-ABC
Revised: SMC-ABC

Original: (Beaumont et al., 2009) (the slightly modiﬁed version) and (ii) SNL (Papamakarios et al., 2019).
Revised: (Beaumont et al., 2009) (the slightly modiﬁed version) and (ii) SNL (Papamakarios et al., 2019).

Original: We compare the performance of the algorithms augmented with our neural statistic (dubbed as SMC-ABC+ and SNL+) to their original versions as well as the versions based on expertdesigned statistics (details presented later; we call the corresponding methods SMC-ABC’ and SNL’).
Revised: We compare the performance of the algorithms augmented with our neural statistic (dubbed as SMC-ABC+ and SNL+) to their original versions as well as the versions based on expertdesigned statistics (details presented later; we call the corresponding methods SMC-ABC’ and SNL’).

Original: We also compare to the sequential neural posterior estimate (SNPE) method 1 which needs no statistic design, as well as the sequential ratio estimate (SRE) method (Hermans et al., 2019) which is closely related to our MI-based method 2 .
Revised: We also compare to the sequential neural posterior estimate (SNPE) method 1 which needs no statistic design, as well as the sequential ratio estimate (SRE) method (Hermans et al., 2019) which is closely related to our MI-based method 2 .

Original: All methods are run for 10 rounds with 1,000 simulations each.
Revised: All methods are run for 10 rounds with 1,000 simulations each.

Original: Evaluation metric .
Revised: Evaluation metric .

Original: To assess the quality of the estimated posterior, we compare the Jensen-Shannon divergence (JSD) between the approximate posterior Q and the true posterior P in each method:
Revised: To assess the quality of the estimated posterior, we compare the Jensen-Shannon divergence (JSD) between the approximate posterior Q and the true posterior P in each method:

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: 10 rounds 0.
Revised: 10 rounds 0.

Original: J S D
Revised: J S D

Original: SMC-ABC
Revised: SMC-ABC

Original: SMC-ABC+
Revised: SMC-ABC+

Original: SNL
Revised: SNL

Original: SNL+ (b) (c)
Revised: SNL+ (b) (c)

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: For the problems we consider in the experiments, the true posterior P is either analytically available, or could be accurately approximated by a standard rejection ABC algorithm (Pritchard et al., 1999) with known low-dimensional sufﬁcient statistic (e.g s ( x ) ∈ Z ) and extensive simulations (e.g 10 6 ).
Revised: For the problems we consider in the experiments, the true posterior P is either analytically available, or could be accurately approximated by a standard rejection ABC algorithm (Pritchard et al., 1999) with known low-dimensional sufﬁcient statistic (e.g s ( x ) ∈ Z ) and extensive simulations (e.g 10 6 ).

Original: 5. R ESULTS
Revised: 5. R ESULTS

Original: We demonstrate the effectiveness of our method in three cases: (a) an Ising model; (b) a Gaussian copula model; (c) an Ornstein-Uhlenbeck process.
Revised: We demonstrate the effectiveness of our method in three cases: (a) an Ising model; (b) a Gaussian copula model; (c) an Ornstein-Uhlenbeck process.

Original: The Ising model does not have an analytical likelihood but the posterior can be approximated accurately by rejection ABC due to the existence of low-dimensional, discrete sufﬁcient statistic.
Revised: The Ising model does not have an analytical likelihood but the posterior can be approximated accurately by rejection ABC due to the existence of low-dimensional, discrete sufﬁcient statistic.

Original: The last two models have analytical likelihoods and hence analytical posteriors.
Revised: The last two models have analytical likelihoods and hence analytical posteriors.

Original: These models cover the cases of graph data, i.i.d data and sequence data.
Revised: These models cover the cases of graph data, i.i.d data and sequence data.

Original: Ising model .
Revised: Ising model .

Original: The ﬁrst model we consider is a mathematical model in statistical physics that describes the states of atomic spins on a 8 × 8 lattice (see Figure 1.a).
Revised: The ﬁrst model we consider is a mathematical model in statistical physics that describes the states of atomic spins on a 8 × 8 lattice (see Figure 1(a)).

Original: Each spin has two states described by a discrete random variable x i ∈ {− 1 , +1 } , and is only allowed to interact with its neighbour.
Revised: Each spin has two states described by a discrete random variable x i ∈ {− 1 , +1 } , and is only allowed to interact with its neighbour.

Original: Given parameters θθθ
Revised: Given parameters θθθ

Original: = { θ 1 , θ 2 }
Revised: = { θ 1 , θ 2 }

Original: , the probability density function of the Ising model is:
Revised: , the probability density function of the Ising model is:

Original: [Equation]
Revised: [Equation]

Original: where (cid:104) i, j (cid:105) denotes that spin i and spin j are neighbours.
Revised: where (cid:104) i, j (cid:105) denotes that spin i and spin j are neighbours.

Original: H is also called the Hamiltonian of the model.
Revised: H is also called the Hamiltonian of the model.

Original: Here, the likelihood function of this model is not analytical due to the intractable normalizing constant Z ( θθθ ) =
Revised: Here, the likelihood function of this model is not analytical due to the intractable normalizing constant Z ( θθθ ) =

Original: (cid:80) x ∈{− 1 , 1 } m · m e − H ( x ; θθθ ) .
Revised: (cid:80) x ∈{− 1 , 1 } m · m e − H ( x ; θθθ ) .

Original: However, sampling from the model by MCMC is possible.
Revised: However, sampling from the model by MCMC is possible.

Original: Note that the sufﬁcient statistic is known for this model: s ∗ ( x ) = { (cid:80) (cid:104) i,j (cid:105) x i x j , (cid:80) i x i } .
Revised: Note that the sufﬁcient statistic is known for this model: s ∗ ( x ) = { (cid:80) (cid:104) i,j (cid:105) x i x j , (cid:80) i x i } .

Original: The true posterior in this model can easily be approximated by a rejection ABC algorithm run with this low-dimensional sufﬁcient statistic and extensive simulations.
Revised: The true posterior in this model can easily be approximated by a rejection ABC algorithm run with this low-dimensional sufﬁcient statistic and extensive simulations.

Original: Here, we assume that θ 2 is known, and the task is to infer the posterior of θ 1 under an uniform prior θ 1 ∼ U (0 , 1 . 5) (in this case the sufﬁcient statistic becomes only 1D: s ∗ ( x )
Revised: Here, we assume that θ 2 is known, and the task is to infer the posterior of θ 1 under an uniform prior θ 1 ∼ U (0 , 1 . 5) (in this case the sufﬁcient statistic becomes only 1D: s ∗ ( x )

Original: = (cid:80) (cid:104) i,j (cid:105) x i x j ).
Revised: = (cid:80) (cid:104) i,j (cid:105) x i x j ).

Original: The true parameters are θθθ ∗ = { 0 . 3 , 0 . 1 } .
Revised: The true parameters are θθθ ∗ = { 0 . 3 , 0 . 1 } .

Original: In Figure 1(c), we investigate whether the proposed statistic could achieve sufﬁciency.
Revised: In Figure 1(c), we investigate whether the proposed statistic could achieve sufﬁciency.

Original: Ideally, if the learned statistic s in our method does recover the true sufﬁcient statistic s ∗ well, the relationship between s and s ∗ should be nearly monotonic
Revised: Ideally, if the learned statistic s in our method does recover the true sufﬁcient statistic s ∗ well, the relationship between s and s ∗ should be nearly this

Original: [Figure]
Revised: [Figure]

Original: 10 rounds 0.
Revised: 10 rounds 0.

Original: J S D
Revised: J S D

Original: SMC-ABC
Revised: SMC-ABC

Original: SMC-ABC+
Revised: SMC-ABC+

Original: SNL
Revised: SNL

Original: SNL+ (b)
Revised: SNL+ (b)

Original: Truth
Revised: Truth

Original: SMC
Revised: SMC

Original: SMC+
Revised: SMC+

Original: SNL
Revised: SNL

Original: SNL+ (c)
Revised: SNL+ (c)

Original: [Figure]
Revised: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]
Revised: [Table]

Original: Here SMC’ and SNL’ utilize the hand-crafted summary statistics guided by human prior knowledge.
Revised: Here SMC’ and SNL’ utilize the hand-crafted summary statistics guided by human prior knowledge.

Original: we plot the relationship between s ∗ and s .
Revised: we plot the relationship between s ∗ and s .

Original: We see from the ﬁgure that s learned in our method does increases monotonically with s ∗ approximately, suggesting that swell recovers s ∗ .
Revised: We see from the ﬁgure that s learned in our method does increase monotonically with s ∗ approximately, suggesting that s recovers s ∗ reasonably well.

Original: As comparison, the statistic learned in the widely-used posterior-mean-as-statistic approach only has weak dependence on the true sufﬁcient statistic; it is nearly indistinguishable for different s ∗ .
Revised: As comparison, the statistic learned in the widely-used posterior-mean-as-statistic approach only has weak dependence on the true sufﬁcient statistic; it is nearly indistinguishable for different s ∗ .

Original: In other words, it loses sufﬁciency.
Revised: In other words, it loses sufﬁciency.

Original: The result also veriﬁes our previous theoretical result in Proposition 2.
Revised: The result also veriﬁes our previous theoretical result in Proposition 2.

Original: Figure 1(b) further shows the JSD between the true and learned posterior for different methods across the rounds (the vertical lines indicates standard errors, each JSD is obtained by calculating the average of 3 independent runs.
Revised: Figure 1(b) further shows the JSD between the true and learned posterior for different methods across the rounds (the vertical lines indicates standard errors, each JSD is obtained by calculating the average of 3 independent runs.

Original: The results shown in the below experiments have the same setup).
Revised: The results shown in the below experiments have the same setup).

Original: It can be seen from the ﬁgure that for this model, likelihood-free inference methods augmented with the proposed statistic (SMC-ABC+, SNL+) outperforms their original counterparts (SMC-ABC, SNL) by a large margin.
Revised: It can be seen from the ﬁgure that for this model, likelihood-free inference methods augmented with the proposed statistic (SMC-ABC+, SNL+) outperform their original counterparts (SMC-ABC, SNL) by a large margin.

Original: In Table 1, we further compare our statistic with the expert designed statistic, from which one can see their close performance (here the expert statistic is taken as the true sufﬁcient statistic s ∗ ).
Revised: In Table 1, we further compare our statistic with the expert designed statistic, from which one can see their close performance (here the expert statistic is taken as the true sufﬁcient statistic s ∗ ).

Original: We also see that our method also outperforms SRE which directly estimates the ratio t ( x ,θθθ )
Revised: We also see that our method also seems to outperform SRE which directly estimates the ratio t ( x ,θθθ )

Original: =
Revised: =

Original: p ( x ,θθθ ) /p
Revised: p ( x ,θθθ ) /p

Original: ( x ) p ( θθθ ) ∝ L ( θθθ ; x ) in high-dimensional space (note that the true likelihood is actually of the form L ( θθθ ; x ) =
Revised: ( x ) p ( θθθ ) ∝ L ( θθθ ; x ) in high-dimensional space (note that the true likelihood is actually of the form L ( θθθ ; x ) =

Original: exp( θθθs ∗ ( x )) /Z
Revised: exp( θθθs ∗ ( x )) /Z

Original: ( θθθ ) ) as well as SNPE (version B).
Revised: ( θθθ ) ) as well as SNPE (version B).

Original: The reason why
Revised: The reason why

Original: SNPE(-B) does not perform more satisfactorily might be due to the fact that it relies on importance weights to facilitate sequential learning, which induces high variance that makes its training unstable.
Revised: SNPE(-B) does not perform more satisfactorily might be due to the fact that it relies on importance weights to facilitate sequential learning, which induces high variance that makes its training unstable.

Original: Gaussian copula .
Revised: Gaussian copula .

Original: The second model we consider is a 2D Gaussian copula model.
Revised: The second model we consider is a 2D Gaussian copula model (Chen & Gutmann, 2019).

Original: Data x in this model can be seen as generated from the latent variable z by the following process:
Revised: Data x in this model can be seen as generated from the latent variable z as follows:

Original: [Equation]
Revised: [Equation]

Original: where Φ( · ) , F 1 ( x 1 ; θ 1 ) , F 2 ( x 2 ; θ 2 ) are the cumulative distribution function (CDF) of standard normal distribution, the CDF of f 1 ( x 1 ; θ 1 ) and the CDF of f 2 ( x 2 ; θ 2 ) respectively.
Revised: where Φ( · ) , F 1 ( x 1 ; θ 1 ) , F 2 ( x 2 ; θ 2 ) are the cumulative distribution function (CDF) of standard normal distribution, the CDF of f 1 ( x 1 ; θ 1 ) and the CDF of f 2 ( x 2 ; θ 2 ) respectively.

Original: We assume that a total number of 200 samples are i.i.d drawn from this model, yielding a population X
Revised: We assume that a total number of 200 samples are i.i.d drawn from this model, yielding a population X

Original: = { x i } 200 i =1 that serves as our observed data.
Revised: = { x i } 200 i =1 that serves as our observed data.

Original: Note that the likelihood of this model can be computed analytically by the law of variable transformation.
Revised: Note that the likelihood of this model can be computed analytically by the law of variable transformation.

Original: To perform inference, we compute a rudimentary statistic to describe X , namely (a) the 20-equally spaced quantiles of the marginal distributions of X and (b) the correlation between the latent variables z 1 , z 2 in X , resulting in a statistic of dimensionality 41.
Revised: To perform inference, we compute a rudimentary statistic to describe X , namely (a) the 20-equally spaced quantiles of the marginal distributions of X and (b) the correlation between the latent variables z 1 , z 2 in X , resulting in a statistic of dimensionality 41.

Original: An uniform prior is set: θ 1 ∼ U (0 . 5 , 12 . 5) , θ 2 ∼ U (0 , 1) , θ 3 ∼ U (0 . 4 , 0 .
Revised: An uniform prior is set: θ 1 ∼ U (0 . 5 , 12 . 5) , θ 2 ∼ U (0 , 1) , θ 3 ∼ U (0 . 4 , 0 .

Original: 8) and θθθ ∗ = { 6 , 0 . 5 , 0 . 6 } .
Revised: 8) and θθθ ∗ = { 6 , 0 . 5 , 0 . 6 } .

Original: In Figure 2.(b), we demonstrate the power of our neural sufﬁcient statistic learning method on the
Revised: In Figure 2(b), we demonstrate the power of our neural sufﬁcient statistic learning method on the

Original: Gaussian copula problem.
Revised: Gaussian copula problem.

Original: Overall, we see that the proposed method signiﬁcantly boosts the accuracy 
Revised: Overall, we see that the proposed method improves the accuracy of

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: J S D
Revised: J S D

Original: SMC-ABC
Revised: SMC-ABC

Original: SMC-ABC+
Revised: SMC-ABC+

Original: SNL
Revised: SNL

Original: SNL+ (b)
Revised: SNL+ (b)

Original: Truth
Revised: Truth

Original: SMC
Revised: SMC

Original: SMC+
Revised: SMC+

Original: SNL
Revised: SNL

Original: SNL+ (c)
Revised: SNL+ (c)

Original: [Figure]

Original: [Table]
Revised: [Table]

Revised: Here SMC’ and SNL’ utilize the hand-crafted summary statistics guided by human prior knowledge.

Original: of existing likelihood-free inference methods, as well asimproving their robustness (see e.g the reduced variability in SNL+.
Revised: existing likelihood-free inference methods, as well as their robustness (see e.g the reduced variability in SNL+.

Original: The high variability in SNL may be due to the lack of training data required to learn the 41-dimensional likelihood function well).
Revised: The high variability in SNL may be due to the lack of training data required to learn the 41-dimensional likelihood function well).

Original: This is also conﬁrmed by the contours plots in Figure 2.(c).
Revised: This is also conﬁrmed by the contours plots in Figure 2(c).

Original: In Table 2 we further compare the proposed statistic with the expert-designed low-dimension statistic (here the expert statistic is taken to be the 5 -equally spaced marginal quantiles + the correlations between z 1 , z 2 ), from which we see that our proposed statistic achieves much better performance.
Revised: In Table 2 we further compare the proposed statistic with the expert-designed low-dimension statistic (here the expert statistic is taken to be the 5 -equally spaced marginal quantiles + the correlations between z 1 , z 2 ), from which we see that our proposed statistic achieves a better performance.

Original: For this model, our method seems to perform slightly worse than SNPE (possibly due to the imperfect infomax learning on this problem) but the gap there is indeed very small ( ≤ 0 .
Revised: For this model, our method seems to perform slightly worse than SNPE (possibly due to the imperfect infomax learning on this problem), but the gap there is indeed very small ( ≤ 0 .

Original:  JSD).
Revised: 005 JSD).

Original: We conjecture that more reliable infomax learning methods might help to resolve this issue.
Revised: We conjecture that more reliable infomax learning methods might help to resolve this issue.

Original: Ornstein-Uhlenbeck process .
Revised: Ornstein-Uhlenbeck process .

Original: The last model we consider is a discreterized stochastic differential equation (SDE).
Revised: The last model we consider is a discretized stochastic differential equation (SDE).

Original: Data x = { x t } Dt =1 in this model is sequentially generated as:
Revised: Data x = { x t } Dt =1 in this model is sequentially generated as:

Original: [Equation]
Revised: [Equation]

Original: where D = 50 , ∆ t = 0 .
Revised: where D = 50 , ∆ t = 0 .

Original: 2 and x 0 = 10 .
Revised: 2 and x 0 = 10 .

Original: This model is Markovian, and has an analytical likelihood.
Revised: This model is Markovian, and has an analytical likelihood.

Original: It has a wide applications in ﬁnancial mathematics and physical sciences.
Revised: It has a wide applications in ﬁnancial mathematics and physical sciences.

Original: Here, the parameters of interest are θθθ = { θ 1 , θ 2 } , and a uniform prior is placed on these parameters:
Revised: Here, the parameters of interest are θθθ = { θ 1 , θ 2 } , and a uniform prior is placed on these parameters:

Original: θ 1 ∼ U (0 , 1) , θ 1 ∼
Revised: θ 1 ∼ U (0 , 1) , θ 1 ∼

Original: U ( − 2 . 0 , 2 . 0) .
Revised: U ( − 2 . 0 , 2 . 0) .

Original: The true parameters are set to be θθθ ∗ = { 0 . 5 , 1 . 0 } .
Revised: The true parameters are set to be θθθ ∗ = { 0 . 5 , 1 . 0 } .

Original: Figure 3(b) compares the JSD of each method against the simulation cost.
Revised: Figure 3(b) compares the JSD of each method against the simulation cost.

Original: Again, we ﬁnd that the proposed neural sufﬁcient statistics greatly improve the performance of both SMC-ABC and SNL.
Revised: Again, we ﬁnd that the proposed neural sufﬁcient statistics greatly improves the performance of both SMC-ABC and SNL.

Original: In
Revised: In

Original: Table 3, we compare our statistics to expert statistic (here the expert statistics are taken as the mean, standard error and autocorrelation with lag 1,2 of the time series).
Revised: Table 3, we compare our statistics to expert statistic (here the expert statistics are taken as the mean, standard error and autocorrelation with lag 1, 2, 3 of the time series).

Original: It can be seen that our statistics are comparable to the expert statistics.
Revised: It can be seen that our statistics are comparable to the expert statistics.

Original: Our method also signiﬁcantly outperforms SRE and SNPE .
Revised: Our method also seems to outperform SRE and SNPE.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We propose a new deep learning-based approach for automatically constructing low-dimensional sufﬁcient statistics in likelihood-free inference.
Revised: We propose a new deep learning-based approach for automatically constructing low-dimensional sufﬁcient statistics in likelihood-free inference.

Original: The obtained neural approximate sufﬁcient statistics can be applied to both traditional ABC-based and recent NDE-based methods.
Revised: The obtained neural approximate sufﬁcient statistics can be applied to both traditional ABC-based and recent NDE-based methods.

Original: The main hypothesis of the approach is that learning such sufﬁcient statistic via the infomax principle might be easier than estimating the density itself.
Revised: The main hypothesis of the approach is that learning such sufﬁcient statistic via the infomax principle might be easier than estimating the density itself.

Original: We verify this hypothesis by experiments on various tasks with graphs, i.i.d and sequence data.
Revised: We verify this hypothesis by experiments on various tasks with graphs, i.i.d and sequence data.

Original: Our method establishes a link between representation learning and likelihood-free inference communities.
Revised: Our method establishes a link between representation learning and likelihoodfree inference communities.

Original: For future works, we can consider other infomax approaches.
Revised: For future works, we can consider more infomax representation learning approaches, as well as methods to automatically determine the dimensionality

Original: A T HEORETICAL RESULTS A. P ROOF OF P ROPOSITION
Revised: A T HEORETICAL PROOFS A. P ROOF OF P ROPOSITION

Original: Proof .
Revised: Proof .

Original: Firstly, assume s ( · ) is a sufﬁcient statistic.
Revised: Firstly, assume s ( · ) is a sufﬁcient statistic.

Original: By the deﬁnition of sufﬁcient statistic we know p ( x | θθθ ) =
Revised: By the deﬁnition of sufﬁcient statistic we know p ( x | θθθ ) =

Original: p ( x | s ) p ( s | θθθ ) .
Revised: p ( x | s ) p ( s | θθθ ) .

Original: Then we have the Markov chain θθθ → s → X for the data generating process.
Revised: Then we have the Markov chain θθθ → s → X for the data generating process.

Original: On the other hand, since x ∼ p ( x | θθθ ) and S is a deterministic function we have the Markov chain θθθ
Revised: On the other hand, since x ∼ p ( x | θθθ ) and S is a deterministic function we have the Markov chain θθθ

Original: → X → S .
Revised: → X → S .

Original: By data processing inequality we have I ( θθθ ; s ( X ))
Revised: By data processing inequality we have I ( θθθ ; s ( X ))

Original: ≤
Revised: ≤

Original: I ( θθθ ; X ) for the ﬁrst chain and
Revised: I ( θθθ ; X ) for the ﬁrst chain and

Original: I ( θθθ ; X ) ≤ I ( θθθ ; s ( X )) for the second chain.
Revised: I ( θθθ ; X ) ≤ I ( θθθ ; s ( X )) for the second chain.

Original: This implies that I ( θθθ ; X ) =
Revised: This implies that I ( θθθ ; X ) =

Original: I ( θθθ ; s ( X ))
Revised: I ( θθθ ; s ( X ))

Original: i.e s is the maximizer of I ( θθθ ; S ( X )) .
Revised: i.e s is the maximizer of I ( θθθ ; S ( X )) .

Original: For the other direction, since I ( θθθ ; s ( X ))
Revised: For the other direction, since I ( θθθ ; s ( X ))

Original: = max S : X→S
Revised: = max S : X→S

Original: I ( θθθ ; S ( X )) , we have I ( θθθ ; s ( X ))
Revised: I ( θθθ ; S ( X )) , we have I ( θθθ ; s ( X ))

Original: =
Revised: =

Original: I ( θθθ ; X ) .
Revised: I ( θθθ ; X ) .

Original: Note that θθθ → X → s is a Markov chain, from Theorem 2.8.1 of
Revised: Note that θθθ → X → s is a Markov chain, from Theorem 2.8.1 of

Original: Cover et al.
Revised: Cover et al.

Original: (2003) we can get θθθ and X is conditionally independent given s .
Revised: (2003) we can get θθθ and X is conditionally independent given s .

Original: This results in that s is sufﬁcient.
Revised: This results in that s is sufﬁcient.

Original: A. P ROOF OF P ROPOSITION
Revised: A. P ROOF OF P ROPOSITION

Original: Proof .
Revised: Proof .

Original: We can rewrite the above objective as E p ( θθθ, x ) [ (cid:107) S ( x ) − θθθ (cid:107) 22 ] = (cid:82) p ( θθθ, x ) log e (cid:107) S ( x )
Revised: We can rewrite the above objective as E p ( θθθ, x ) [ (cid:107) S ( x ) − θθθ (cid:107) 22 ] = (cid:82) p ( θθθ, x ) log e (cid:107) S ( x )

Original: − θθθ (cid:107) 

Revised: − θθθ (cid:107)

Original: d x dθθθ .
Revised: d x dθθθ .

Original: On the other hand we have I ( θ θ θ, S ( x ))
Revised: On the other hand we have I ( θ θ θ, S ( x ))

Original: = (cid:82) p ( θ θ θ, x ) log p ( S ( x ) | θ θ θ ) /p
Revised: = (cid:82) p ( θ θ θ, x ) log p ( S ( x ) | θ θ θ ) /p

Original: ( S ( x ))
Revised: ( S ( x ))

Original: d x dθ θ θ .
Revised: d x dθ θ θ .

Original: By comparing them it is easy to see that they are generally not equivalent except for very rare case (e.g Gaussian).
Revised: By comparing them it is easy to see that they are generally not equivalent except for very rare case (e.g Gaussian).

Original: B M ORE EXPERIMENTAL DETAILS AND RESULTS B. D ETAILED EXPERIMENTAL SETTINGS
Revised: B M ORE EXPERIMENTAL DETAILS AND RESULTS B. D ETAILED EXPERIMENTAL SETTINGS

Original: Neural networks settings .
Revised: Neural networks settings .

Original: For the statistic network, we adopt a D -100-100-
Revised: For the statistic network S in our method, we adopt a D -100-100-

Original: d fully-connected architecture with D being the dimensionality of input data and d the dimensionality of the statistic.
Revised: d fully-connected architecture with D being the dimensionality of input data and d the dimensionality of the statistic.

Revised: For the network H used to extract the representation of θθθ , we adopt a D -100-100-

Revised: K fully-connected architecture with K being the dimensionality of the model parameter.

Revised: For the critic network, we adopt a ( d + K ) -100-1 fully connected architecture with K being the dimensionality of θθθ .

Original: ReLU is adopted as the non-linearity in s ( · ) .
Revised: ReLU is adopted as the non-linearity in all networks.

Original: For the critic network, we adopt a ( d + K ) -100fully connected architecture with K being the dimensionality of θθθ .These neural networks are trained with Adam (Kingma & Ba, 2014) with a learning rate of 1 × 10 − 3 and a batch size of 100.
Revised: For SRE, which is closely related to our method, we use a ( D + K ) -144-144-100-1 architecture. This architecture has a similar complexity as our networks. All these neural networks are trained with Adam (Kingma & Ba, 2014) with a learning rate of 1 × 10 − 3 and a batch size of 100.

Original: No weight decay is applied.
Revised: No weight decay is applied.

Original: For the neural density estimator in SNL, which is realized by a Masked Autoregressive Flow (MAF) (Papamakarios et al., 2017), we adopt 5 autoregressive layers, each of which has two hidden layers with 50 tanh units.
Revised: For the neural density estimator in SNL/SNPE, which is realized by a Masked Autoregressive Flow (MAF) (Papamakarios et al., 2017), we adopt 5 autoregressive layers, each of which has two hidden layers with 50 tanh units.

Original: This is the same settings as in SNL.
Revised: This is the same settings as in SNL.

Original: The MAF are trained with Adam(Kingma & Ba, 2014) with a learning rate of 5 × 10 − 4 and a batch size ofand a slight weight decay ( 1 e − 5 ).
Revised: The MAF are trained with Adam with a learning rate of 5 × 10 − 4 and a batch size of 500 and a slight weight decay ( 1 e − 5 ).

Original: We take 20% of the data for validation, and stop training if the validation error does not improve after 100 epochs 4 .
Revised: For all networks, we take 20% of the data for validation, and stop training if the validation error does not improve after 100 epochs.

Revised: We take the snapshot with the best validation error as the result.

Revised: Sampling from the approximate posterior .

Revised: For fair comparison, we adopt simple rejection sampling for all LFI methods (ABC, SNL, SNPE, SRE) when sampling from the learnt posterior, so that each LFI algorithm only differs in the way they learn the posterior.

Original: Dimensionality of the learned statistic .
Revised: Dimensionality of the learned statistic .

Original: For the JSD/contour plots presented in the experiment part in the main text, we use d = (cid:98) D/ 5 (cid:99) , (cid:98) D/ 5 (cid:99) ,
Revised: For the JSD/contour plots presented in the experiment part in the main text, we use d = (cid:98) D/ 5 (cid:99) , (cid:98) D/ 5 (cid:99) ,

Original: (cid:98) D/
Revised: (cid:98) D/

Original: 10 (cid:99) for the Ising model problem, Gaussian copula problem and OP process problem respectively (remark that D is the dimensionality of the raw data).
Revised: 10 (cid:99) for the Ising model problem, Gaussian copula problem and OP process problem respectively (remark that D is the dimensionality of the raw data).

Original: Training of statistic network .
Revised: Training of statistic network .

Original: Remark that the statistic network S ( · ) is trained with the following objective together with the scoring network T ( · ) :
Revised: Remark that the statistic network S ( · ) is trained with the following objective together with the scoring network T ( · ) :

Original: [Equation]
Revised: [Equation]

Original: the mini-batch approximation to this objective is:
Revised: the mini-batch approximation to this objective is:

Original: [Equation]
Revised: [Equation]

Original: where { j 1 , j 2 , ..., j n } is the j -th random permutation of the indexes { 1 , 2 , ..., n } and the pair ( θθθ i , x i ) are randomly picked from the data D
Revised: where { j 1 , j 2 , ..., j n } is the j -th random permutation of the indexes { 1 , 2 , ..., n } and the pair ( θθθ i , x i ) are randomly picked from the data D

Original: = { θθθ i , x i } Ni =1 .
Revised: = { θθθ i , x i } N i =1 .

Original: Here we simply set m = 200 and n = 100 .
Revised: Here we simply set m = 200 and n = 100 .

Original: JSD calculation between true posterior and approximate posterior .
Revised: JSD calculation between true posterior and approximate posterior .

Original: The calculation of the JensenShannon divergence between true posterior P and approximate posterior Q namely JSD ( P, Q ) = 12 KL [ P (cid:107) Q ] +  12 KL [ Q (cid:107)  P ] is done numerically by a Riemann sum over 30 K equally spaced grids with K being the dimensionality of θθθ .
Revised: The calculation of the JensenShannon divergence between true posterior P and approximate posterior Q namely JSD ( P, Q ) = 12 KL [ P (cid:107) ( P + Q ) / 2]+ 12 KL [ Q (cid:107) ( P + Q ) / 2] is done numerically by a Riemann sum over 30 K equally spaced grids with K being the dimensionality of θθθ .

Original: The region of these grids is jointly deﬁned by 500 samples drawn from P and Q .
Revised: The region of these grids is jointly deﬁned bysamples drawn from P and Q .

Original: When we are only equipped with samples of the true posterior (e.g the Ising model problem), we estimate P by KDE (remark that for the Ising model the posterior is 1D).
Revised: When we are only equipped with samples of the true posterior (e.g the Ising model problem), we estimate P by KDE (remark that for the Ising model the posterior is 1D).

Original: B. A DDITIONAL EXPERIMENTAL RESULTS Comparison of different MI estimators
Revised: B. A DDITIONAL EXPERIMENTAL RESULTS Comparison of different MI estimators

Original: [Figure]
Revised: [Figure]

Original: We conduct empirical study on a 10-round Gaussian copula task with four different mutual information estimators: MINE (Belghazi et al., 2018), MINE-
Revised: We conduct empirical study on a 10-round Gaussian copula task with four different mutual information estimators: MINE (Belghazi et al., 2018), MINE-

Original: f (Belghazi et al., 2018), Jensen-Shannon divergence (JSD) estimators (Hjelm et al., 2018), and classical ratio trick method (Cranmer et al., 2015; Thomas et al., 2016).
Revised: f (Belghazi et al., 2018), Jensen-Shannon divergence (JSD) estimators (Hjelm et al., 2018), and classical ratio trick method (Cranmer et al., 2015; Thomas et al., 2016).

Original: The results are averaged over 3 independent runs.
Revised: The results are averaged over 3 independent runs.

Original: We use the JSD between the true posterior and approximate posterior as in the main text as the metric.
Revised: We use the JSD between the true posterior and approximate posterior as in the main text as the metric.

Original: From Figure 4 we see the JSD estimator achieves the best performance, and as such we adopt it as the MI estimator in our method.
Revised: From Figure 4 we see the JSD estimator achieves the best performance, and as such we adopt it as the MI estimator in our method.

Original: Results under MMD metric .
Revised: Results under MMD metric .

Original: We also conduct experiments with MMD (Gretton et al., 2012) discrepancy on Gaussian copula and OU process with different reruns as a reference.
Revised: For reference, we also conduct experiments with MMD (Gretton et al., 2012) discrepancy on Gaussian copula and OU process with different reruns as a reference.

Original: Our proposed methods achieve consistent performance under JSD metric. The results are averaged over 3 independent runs.
Revised: Our proposed methods achieve consistent performance under JSD metric (we remind the reader here that the result by JSD should be more reliable since it directly compares two analytical densities)

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: On directly extracting the posterior from the MI network .
Revised: On directly extracting the posterior from the MI network .

Original: As mentioned in the main text, it has been shown that the optimal critic t ( θθθ, s ) in f -divergence-based MI estimator I ( θθθ, s ) indeed corresponds to the density ratio up to a constant: p ( θθθ, s ) /p
Revised: As mentioned in the main text, it has been shown that the optimal critic t ( θθθ, s ) in f -divergence-based MI estimator I ( θθθ, s ) indeed corresponds to the density ratio up to a constant: p ( θθθ, s ) /p

Original: ( θθθ ) p ( s ) .
Revised: ( θθθ ) p ( s ) .

Original: In JSD-based MI estimator, which we used to learn the sufﬁcient statistics, we can extract the ratio as
Revised: In JSD-based MI estimator, which we used to learn the sufﬁcient statistics, we can extract the ratio as

Original: [Equation]
Revised: [Equation]

Original: with which we can recover the (sufﬁcient statistics-based) posterior estimate as:
Revised: with which we can recover the (sufﬁcient statistics-based) posterior estimate as:

Original: [Equation]
Revised: [Equation]

Original: So one might wonder how the posterior estimated in this strategy is compared to the strategy mentioned in the main text (namely re-ﬁtting a neural density estimator q ( s | θθθ ) on s ).
Revised: So one might wonder how the posterior estimated in this strategy is compared to the strategy mentioned in the main text (namely re-ﬁtting a neural density estimator q ( s | θθθ ) on s ).

Original: The ﬁgures below show the results.
Revised: The ﬁgures below show the results.

Revised: [Equation]

Original: 0.7 0.8 θ 0 0. 
Revised: 0.8 θ 0 0. θ0.3 0.4 0.

Revised: [Equation]

Original: θ0.3 0.4 0.5 0.6 0.7 0.8 θ 0 0. 
Revised: 0.6 0.7 0.8 θ 0 0. θ0. [Equation]

Original: θ0.3 0.4 0.5 0.6 0.7 0.8 θ 0 0. 
Revised: [Equation] 0.8 θ 0 0. [Equation]

Original: θtruth extract from T
Revised: θtruth extract from T

Original: NDE reﬁt
Revised: NDE reﬁt

Original: [-0.7 0.8 θ 0 0. θ0.1 0.2 0.3 0.4 0.5 0.
-]
Revised: [-0.7 0.8 θ 0 0. θ0.1 0.2 0.3 0.4 0.5 0.
-][Equation]

Original: 0.7 0.8 θ 0 0.
Revised: 0.8 θ 0 0.

Original: θ0.1 0.2 0.3 0.4 0.5 0.

Revised: θ0.1 0.2 0.3 0.4 0.5 0.

Original: 0.7 0.8 θ 0 0.
Revised: 0.7 0.8 θ 0 0.

Original: θtruth extract from T
Revised: θtruth extract from T

Original: NDE reﬁt
Revised: NDE reﬁt

Original: We discover that the posteriors returned by NDE reﬁt are generally more accurate than the one directly extracted from the critic T .
Revised: We discover that the posteriors returned by NDE reﬁt seem to be more accurate than the one directly extracted from the critic T .

Original: Here, the JSD in the two problems are (0.000, 0.014, 0.004) and (0.000, 0.010, 0.004) respectively (the OU process is the same as in the main text.
Revised: Here, the JSD in the two problems are (0.000, 0.014, 0.004) and (0.000, 0.010, 0.004) respectively (the OU process is the same as in the main text.

Original: The wrapped Gaussian  is one where the data is generated as x ∼ N ( x , tanh ( θθθ ) , I ) .
Revised: The wrapped Gaussian model is one where the data is generated as x ∼ N ( x , tanh ( θθθ ) , I ) .

Original: All results are obtained with 10, simulations).
Revised: All results are obtained with 10,000 simulations).

Original: More speciﬁcally,  the NDE posteriors are smoother and hence closer to a realistic pdf.
Revised: More speciﬁcally, we discover that the NDE posteriors tends to be smoother and hence closer to a realistic pdf.

Revised: We conjecture that this might due to two reasons:

Revised: • In NDE reﬁt, the density is guaranteed to be a valid pdf during the learning procedure.

Revised: This might be a good inductive bias that helps to learn the density better.

Original: This might be due to that the critic only learns the unnormalized ratio, and the posterior recovered from this ratio  is not guaranteed to be pdf.
Revised: The density extracted from the ratio on the contrary is not guaranteed to be a valid one, so it is less realistic; • It is also possible that even if the whole statistics + critic networks does not estimate the density perfectly, the statistics network itself might still do a good job in learning the sufﬁcient statistics. This opinion was also

Revised: appreciated

Revised: in

Original: The NDE-based strategy, on the contrary, utilizes the inductive bias that the posterior must be correctly normalized, hence is more close to the truth.
Revised: previous works (see e.g Cranmer et al. Nonetheless we feel more exploration is required to draw a conclusion.

Revised: We leave it to future work.

Revised: Contrastive learning v.s MLE .

Revised: In the experiment in the main text, we discover that our method does not always achieve the best performance: it loses to SNPE-B on the Gaussian.

Revised: Here we would like to investigate why this happens.

Revised: Upon a closer look, we discover that SRE, which is closely related to our method, is also outperformed by SNPE-B on the Gaussian copula problem.

Revised: Both SRE and our method uses contrastive learning but not MLE during learning.

Revised: Since both of these two contrastive learning methods are outperformed by the MLE-based SNPE-B, it makes us suspect the reason is due to imperfect contrastive learning.

Revised: To verify this, we further conduct experiments for SNPE-C, which shares the same loss function with SRE but with a different parameterization to the density ratio (SRE: fully-connected network;

Revised: SNPE-C: NDE-based parameterization.

Revised: This NDE is the same as in SNL).

Original: This also resonates with our hypothesis: learning the sufﬁcient statistic needs not to know the exact density.
Revised: The result is as below: [Table] Surprisingly, we ﬁnd that like SRE and SNL+, SNPE-C also perform less satisfactorily than SNPE-B on the Gaussian copula problem. This suggests that contrastive learning might be less preferable than

Revised: MLE on this problem, which this might explain the less satisfactory performance of our method.