Towards Reliable Simulation-Based Inference with Balanced Neural Ratio Estimation
Abstract
Modern approaches for simulation-based inference rely upon deep learning surrogates to enable approximate inference with computer simulators.
In practice,the estimated posteriors’ computational faithfulness is, however, rarely guaranteed.
For example, Hermans et al.
[1] show that current simulation-based inference algorithms can produce posteriors that are overconﬁdent, hence risking false inferences.
In this work, we introduce Balanced Neural Ratio Estimation ( BNRE ), a variation ofthe NRE algorithm
[2] designed to produce posterior approximations that tend to bemore conservative, hence improving their reliability, while sharing the same Bayesoptimal solution.
We achieve this by enforcing a balancing condition that increasesthe quantiﬁed uncertainty in small simulation budget regimes while still convergingto the exact posterior as the budget increases.
We provide theoretical argumentsshowing that BNRE tends to produce posterior surrogates that are more conservativethan NRE ’s. We evaluate BNRE on a wide variety of tasks and show that it producesconservative posterior surrogates on all tested benchmarks and simulation budgets.
Finally, we emphasize that BNRE is straightforward to implement over NRE anddoes not introduce any computational overhead.
Introduction
Many areas of science and engineering use parametric computer simulations to describe complexstochastic generative processes.
In this setting, Bayesian inference provides a principled frameworkto identify parameters matching empirical observations.
Computer simulations, however, deﬁnethe necessary likelihood function only implicitly, which prevents its evaluation and the use ofclassical inference algorithms.
To overcome this obstacle, recent simulation-based inference (SBI)algorithms [3] build upon deep learning surrogates to approximate parts of the Bayes rule andenable approximate inference.
For example, [4, 5] build a surrogate of the likelihood function while[6, 7, 2, 8, 9] approximate the likelihood-to-evidence ratio.
The posterior can also be targeted directlywith variational inference, as proposed by [10, 11, 5].
These algorithms can each be made amortizedor be run sequentially to drive the training towards a target observation and improve the simulationefﬁciency of the procedure [10, 12, 2, 11, 4, 8, 5].
However, sequential methods have the drawback ofbeing computationally expensive to diagnose as the surrogates are only valid for the target observation[1].
Truncated marginal neural ratio estimation [9] alleviates this issue by introducing a sequentialalgorithm that builds a surrogate valid in a local region around the target.
Since modern simulation-based inference algorithms rely on deep learning surrogates, concernsnaturally arise regarding their computational faithfulness and whether they are sufﬁciently adequatefor the inference task of interest.
In Bayesian inference, these concerns can be at least partiallyaddressed with diagnostics designed to probe the correct behaviour of the inference method, suchas ˆ R diagnostics for MCMC [13], or to assess the quality of posterior approximations directly.
The latter include diagnostics such as simulation-based calibration (SBC) [14] or coverage-baseddiagnostics [15, 1].
As discussed by Hermans et al.
[1], posterior approximations must be conservativeto guarantee reliable and meaningful inferences, even when approximations are not faithful.
Forexample, in the physical sciences, where the goal is often to constrain parameters of interest, wronglyexcluding plausible values could drive the scientiﬁc inquiry in the wrong direction, whereas failingto exclude implausible values because of (too) conservative estimations is much less detrimental.
Unfortunately, the same authors also demonstrate that current simulation-based inference algorithmscan lead to overconﬁdent surrogates and therefore false inferences.
In this work, we develop a novelalgorithm that not only converges to exact inference as the simulation budget increases, but which isalso more likely to produce conservative surrogates in small simulation budget regimes.
Towardsthis objective, we propose a variant of the NRE algorithm called Balanced Neural Ratio Estimation( BNRE ), which enforces a balancing condition on the binary neural classiﬁer to increase the reliabilityof its posterior approximations.
The structure of the manuscript is outlined as follows.
Section 2 describes the formalism and thenecessary background.
Section 3 describes BNRE and provides theoretical arguments towards itsconservativeness and reliability.
Section 4 illustrates our main results and provides insights regardingthe behaviour of the method.
Finally, Section 5 discusses related work while Section 6 summarizes ourcontributions and hints at future work.
Code is available at github.com/anonymous/anonymous .
Background
2.1 Statistical formalism
This work is concerned with simulation-based inference algorithms that produce posterior approximations ˆ p ( ϑ | x ) under the following semantics.
Target parameters ϑ denote the parameters ofthe model
and we make the reasonable assumption that the prior p ( ϑ ) is tractable.
The model isgenerically expressed as a computer program, a simulator, that describes the forward dynamics ofinterest based on the input parameters ϑ .
The simulator implicitly deﬁnes the likelihood functionp ( x | ϑ ) .
While we cannot directly evaluate the density p ( x | ϑ ) , we can execute the computerprogram to generate synthetic observables x ∼ p ( x | ϑ ) .
Every observable x o is tied to groundtruth parameters ϑ ∗ whose forward evaluation within the simulator produced x ∗ .
Of special importance to Bayesians is the notion of a credible region , which is a domain Θ withinthe target parameter space that satisﬁes
[Equation]
for some observable x ∗ andconﬁdence level 1 − α .
Because many such regions exist, we target the credible region with thesmallest volume, also known as the highest posterior density region [16, 17].
2.2 Neural ratio estimation
Neural Ratio Estimation ( NRE ) is an established approach in the simulation-based inference literatureboth from frequentist [6] and Bayesian [7, 2, 8, 9] perspectives.
In essence, all protocols rely onthe density-ratio trick [18, 19, 6] to construct a surrogate of a likelihood ratio.
In this work, weconsider an amortized estimator ˆ r ( x | ϑ ) of the intractable likelihood-to-evidence ratio r
( x | ϑ ) =
p ( ϑ , x ) /p
( ϑ ) p ( x ) =
p ( x | ϑ ) /p
( x ) that can be learned by training a binary classiﬁer to distinguishbetween samples of the joint p ( ϑ , x ) with class label 1 and samples of the product of marginalsp ( ϑ ) p ( x ) with class label 0.
For the binary cross-entropy loss, the Bayes optimal classiﬁer is
[Equation]
where σ ( · ) is the sigmoid function.
Given target parameters ϑ and an observable x supported by p ( ϑ )and p ( x ) respectively, the learned classiﬁer ˆ d provides an approximation for the log likelihood-toevidence ratio log r ( x | ϑ ) because log r ( x
[Equation]
Balanced binary classiﬁcation for neural ratio estimation
Following Hermans et al.
[1], let us ﬁrst deﬁne the expected coverage probability of the 1 − αhighest posterior density regions derived from the posterior estimator
ˆ p ( ϑ | x ) as
[Equation]
where the function Θ ˆ p ( ϑ | x ) (1 − α ) yields the 1 − α highest posterior density region of ˆ p ( ϑ | x ) .
This diagnostic probes the conservativeness of the posterior estimator (or the lack thereof) and can beinterpreted as the expected frequentist coverage E p (
[Equation]
In this work, a posterior estimator has coverage at the conﬁdence level 1 − α whenever the expectedcoverage probability is larger or equal to the nominal coverage probability, 1 − α .
We say that aposterior estimator is conservative when it has coverage for all conﬁdence levels.
The expectedcoverage probability can be plotted for various levels α , which allows to visually identify conservativeposterior estimators.
The expected coverage can also be shown to be a special case of the SBCdiagnostic [14] (see Appendix A), further motivating the usage of expected coverage.
Our main objective is to restrict the hypothesis space of the approximate classiﬁers ˆ d to those leadingto conservative posterior estimators, hence solving the reliability concerns of NRE .
Towards this goal,we construct a hypothesis space of balanced classiﬁers and show both theoretically and empiricallythat they lead to posterior estimators that tend to be more conservative.
3.1 Balanced binary classiﬁcation
Deﬁnition 1.
A classiﬁer ˆ d is
[Equation]
Theorem 1.
Any balanced classiﬁer ˆ d satisﬁes E p ( ϑ , x )
[Equation]
Proof.
The integral form of the balancing condition
[Equation]
implies that (cid:0) p ( x , ϑ ) + p ( ϑ ) p ( x ) (cid:1)
ˆ d ( ϑ , x ) is a valid density, both integrating to 1 and positiveeverywhere.
Therefore, its Kullback-Leibler (KL) divergence with p ( ϑ , x ) is positive.
Through
Jensen’s inequality, we obtain
[Equation]
Theorem 2.
Any balanced classiﬁer ˆ d satisﬁes E p ( ϑ
[Equation]
[Equation]
Proof.
Similar to Theorem 1, see Appendix B.
Theorem 1 shows that, in expectation over the joint distribution p ( ϑ , x ) , a balanced classiﬁer ˆ dtends to make predictions whose probability values ˆ d ( ϑ , x ) are smaller than the exact probabilityvalues d
( ϑ , x ) .
In other words, a balanced classiﬁer ˆ d tends to be less conﬁdent than the Bayesoptimal classiﬁer d .
Similarly, Theorem 2 shows that, in expectation over the product of the marginalsp ( ϑ ) p ( x ) , a balanced classiﬁer tends to make predictions whose probability values 1 − ˆ d ( ϑ , x ) aresmaller than the exact probability values 1 − d ( ϑ , x ) , hence showing that a balanced classiﬁer ˆ dtends to also be less conﬁdent than the Bayes optimal classiﬁer d .
We note however that these twotheorems hold only in expectation, which implies that neither ˆ d ( ϑ , x ) ≤ d ( ϑ , x ) for all ϑ , x nor [Equation] ˆ [Equation]
[Equation]
for all ϑ , x can generally be guaranteed.
Theorem 3.
The Bayes optimal classiﬁer d ( ϑ , x ) is balanced.
Proof.
Replacing the Bayes optimal classiﬁer
[Equation]
5) in the integral form of the balancing condition, we have
[Equation]
Theorem 3 states that the Bayes optimal classiﬁer is balanced.
Therefore, restricting the modelhypothesis space to balanced classiﬁers does not modify the global optimum .
3.2 Balanced neural ratio estimation
We now extend the NRE algorithm to enforce the balancing condition.
On the one hand, the previousresults show that enforcing the condition should result in increasingly conservative classiﬁers ˆ d andtherefore to dispersed posterior approximations.
Ideally, whenever ˆ d ( ϑ , x ) ≤ d ( ϑ , x ) then [Equation]
d [Equation] ϑ , x ) , which is equivalent to ˆ r
[Equation]
x | ϑ )
[Equation]
r
[Equation]
x | [Equation] [Equation] 6) and ˆ p ( ϑ | x ) ≤ p ( ϑ | x ) since ˆ p ( ϑ | x ) = p ( ϑ )ˆ r ( x | ϑ ) .
On the other hand, the classiﬁer ˆ d may notbe better than a random classiﬁer, which results in probability values ˆ d ( ϑ , x ) = 0 .
5 for all ϑ , x .
Inthat case, ˆ r ( x | ϑ ) {+= 
 and the approximate posterior ˆ+} p ( ϑ | x ) degenerates to the prior p ( ϑ ) .
Overall,imposing the balancing condition should therefore result in approximate posteriors that lie betweenthe prior and the exact posterior, without being more conﬁdent than they should.
Practically, the balancing condition can be targeted through a regularization penalty.
For the binarycross-entropy L [Equation] p [Equation]
log [Equation] ˆ d
[Equation]
ϑ
[Equation] x )) [Equation] and given that thebalancing condition only depends on samples from p ( x ) p ( ϑ ) and p ( x , ϑ ) , the full loss functionalincluding the balancing condition can be expressed as
[Equation]
where λ is a (scalar) hyper-parameter controlling the strength of the balancing condition’s contribution.
The training procedure is summarized in Algorithm 1.
Since the balancing condition needs to be 0for a classiﬁer to be balanced, the hyper-parameter controlling the strength of the balancing conditioncould, in principle, be set arbitrarily large.
However, as the balancing condition is estimated via
Monte Carlo sampling, setting λ to a large value could impair the classiﬁer’s learning ability.
Wefound that λ
= 100 works well across many problem domains with varying simulation budgets.
Experiments
We start by providing an extensive validation of BNRE on a broad range of benchmarks demonstrating that the proposed method alleviates the problem.
Section 4.2 follows up with an illustrativedemonstration on the behaviour of BNRE and its hyper-parameters.
4.1 Extensive validation
Setup We evaluate the expected coverage of posterior estimators produced by both NRE and BNREon various problems, whose descriptions can be found in Appendix C.
The architectures and hyperparameters used for each problem are deﬁned in Appendix D. Our evaluation considers simulationbudgets of increasing size, ranging from 2 10 = 1024 to 2 17 = 131 , 072 samples, and credibilitylevels from 0 .
05 to 0 .
For every simulation budget, we train 5 posterior estimators for 500 epochsand determine the credible region by evaluating the approximated posterior density function in adiscretized and empirically normalized grid of the parameter space with sufﬁcient resolution.
Thesubsequent credible region is the set of parameters whose estimated (and normalized) posteriordensity is higher or equal to an inclusion threshold ﬁtted to obtain the desired credibility level 1 − α .
Details on this procedure are described in Appendix E.
The expected coverage probability is estimatedon 10000 unseen samples from the joint p ( ϑ , x ) , for each considered credibility level.
Expected coverage The expected coverage curves and their interpretation are detailed in Figure 1.
We observe that NRE often produces posterior estimators that are overconﬁdent, especially for smallsimulation budgets.
However, NRE ’s reliability increases with the availability of training data.
Bycontrast, BNRE produces posterior estimators that are conservative on all benchmarks for allsimulation budgets .
Figure 2 explores the same phenomena, highlighting the effect of the simulationbudget, by computing the integrated signed area between the expected coverage curve and thediagonal of a particular simulation which we call Coverage AUC .
From this quantity it is evident thereis a clear distinction between NRE and BNRE with respect to the available simulation budget.
Bothmethods have the tendency to converge towards 0, indicating both methods are moving closer to the
Bayes optimal classiﬁer.
However, the difference between these methods lies with how this solutionis approached.
While NRE can approach this limit from both sides, BNRE consistently producescoverage AUC’s above 0, corresponding to conservative posterior approximations, and thereforeexhibits the desired behaviour (in expectation). Statistical performance In addition to the reliability of the posteriors, we evaluate and compare thestatistical performance of the posterior approximations produced by NRE and BNRE . We estimate theexpected approximate log posterior density E p ( ϑ , x ) (cid:2) log ˆ p ( ϑ | x ) (cid:3) over a large number of pairs ϑ , x ,which captures how well the posterior surrogates ˆ p ( ϑ | x ) approximate the true posteriors p ( ϑ | x )since E p ( ϑ , x ) [log ˆ p ( ϑ | x )]
= − E p ( x ) KL [ p ( ϑ | x [Equation] || ˆ p ( ϑ | x )] [Equation]
Figure 3 shows our results. We observe that enforcing the balancing condition for λ = 100 isassociated with a loss in statistical performance. However, the loss in statistical performance is [Figure]
[Figure]
larger thanand an overconﬁdent estimator smaller than 0 .
We observe that while NRE can produce coverage
AUC both below or above 0 , BNRE always produces a coverage AUC larger than 0 , implying that its posterior approximations are conservative on average.
The means over 5 runs are reported.
A complete overview, including standard deviations, are provided in Appendix F. eventually recovered by increasing the simulation budget.
In fact, practitioners might be inclinedto favor reliability over statistical performance [1] and would therefore be willing to cover this cost. Nevertheless, it is possible to improve the statistical performance by tuning the surrogate, or byincreasing the available simulation budget as we have demonstrated. 4.2 In-depth analysis
In this section, we consider the Weinberg benchmark as described in Appendix C.
The quality ofthe posterior approximations produced by BNRE is initially discussed with respect to the simulationbudget. Afterwards, the effects of the hyper-parameter λ are studied.
SLCP
Weinberg
Spatial
[Figure]
M/G/
[Figure]
Lotka Volterra
Gravitational
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
budget represented by the colormap.
The bias and variance are represented on the right side of the plot.
BNRE is run with λ = 100 .
Consistent with our previous observations in Figure , we observe that the gap in both bias and variance reduces as the simulation budget increases.
Futhermore, in contrast with NRE , the posterior approximations of BNRE are tending towards being increasingly calibrated while at the same time being conservative.
Solid lines represent the mean over 5 runs and shaded areas represent the standard deviation
Quality assessment Because the expected coverage does not capture the quality of an approximation in terms of information gain, we complement our assessment with a bias and variance analysisof the posterior approximations.
Let us consider the expected squared error over the approximateposterior E ˆ p ( ϑ | x ) (cid:104) ( ϑ − ϑ ∗ ) 2 (cid:105) .
With ¯ ϑ ( x ) =
E ˆ p ( ϑ | x ) [ ϑ ] , we decompose E ˆ p ( ϑ | x ) (cid:104) ( ϑ − ϑ ∗ ) 2 (cid:105) as
[Equation]
The expectation over the joint distribution p ( ϑ ∗ , x ) of the expected squared error can hence bedecomposed in a bias term deﬁned as
[Equation]
[Equation]
which can be interpreted as the expected discrepancy between the nominal value ϑ ∗ and the expectedposterior value ¯ ϑ .
The variance term is
[Equation]
2 ϑ 0 .0 .1 .
E s t i m a t e d p o s t e r
i o r d e n s i t y
Eﬀect on posteriors 0 .0 .1 .
Credibility level 0 .0 .0 .0 .0 .1 .
E x p ec t e d c o v e r ag e
Eﬀect on expected coverage 1010 3 Hyper-parameter λ 0 .0 .0 .0 .
C o v e r ag e AU
C
{+Eﬀect on coverage AUC 1010 3 Hyper-parameter λ 0 .0 .0 .0 .0 .
+}
.| 1 −
B [ ˆ d ] |
Eﬀect on B [ˆ d ]
[Figure]
and measures the dispersion of the posterior approximations.
Note that these terms differ fromthe typical statistical bias and variance of point estimators since we are considering full posteriorestimators.
In particular, in our case, the bias of the Bayes optimal model does not necessarily reduceto 0 .
Figure 4 shows the evolution of expected coverage, bias and variance with respect to the availablesimulation budget.
By taking all plots into consideration with respect to the simulation budget, wecan validate that – as suggested by theorems 1 and 2 – the increase in expected coverage is tied to anincrease in variance.
However, this increase comes at the price of a slight increase in bias.
Consistentwith our previous observations in Figure 3, we observe that the gap in both bias and variance reducesas the simulation budget increases.
A bias and variance analysis for all remaining benchmarks isdiscussed in Appendix G.
Effects of λ Finally, Figure 5 shows the effect the hyper-parameter λ on the posterior approximations, their expected coverage and the balancing condition.
BNRE is run 5 times for λ ranging from 1to 2 15 and for a ﬁxed simulation budget of 1024 .
Initially, the effect on the posterior approximations islimited for small values of λ .
However, once λ increases, the balancing condition forces the posteriorapproximations to become increasingly dispersed and conservative.
Eventually, at least for thisspeciﬁc simulation budget, the posterior approximation reduces to the prior as the balancing conditionbecomes dominant over the cross-entropy term.
Although the global optimum remains unchangedas stated by Theorem 3, large λ values are likely to impair the training procedure.
In particular, alarge λ can inﬂate the statistical noise of the Monte Carlo estimation of the balancing conditionand make the classiﬁer ˆ d degenerate to a classiﬁer that is trivially balanced such as the randomclassiﬁer ˆ d ( ϑ , x ) = 0 .
5 for all ϑ , x .
In this case, ˆ r ( x | ϑ ) = 1 for all ϑ , x and the approximateposterior degenerates to the prior.
This effect is directly evident from Figure 5, starting from λ (cid:39) 1000 .
In practice, λ should be sufﬁciently large such that the approximate classiﬁer is balanced, whilemaximizing the statistical performance of the posterior estimator.
We found λ = 100 to perform wellacross all benchmarks, which again, is supported by Figure 5.
Related work
In the Bayesian setting, BNRE improves the reliability of NRE by constraining the classiﬁer hypothesisspace to balanced classiﬁers, which results in more conservative posteriors.
Towards the same objective of conservative and reliable approximate posteriors, Hermans et al.
[1] have shown empirically that ensembling posterior estimators increases their expected coverage.
Since the two solutions arecomplementary, we suggest that ensembling BNRE is a safe practice to follow.
To the best of ourknowledge, no other related work exists to make Bayesian simulation-based inference algorithmsmore conservative and reliable.
In the frequentist setting, Cranmer et al.
[6] make use of neural ratio estimation to learn likelihoodratio test statistics.
They show that the classiﬁer ˆ d does not need to be exact for the statistic to remainthe most powerful, provided that the approximate likelihood ratio is monotonic with exact likelihoodratio.
When this is not the case, robust inference remains possible by calibrating the classiﬁer’soutput, at the price of a loss in statistical power.
Similarly, for frequentist likelihood-free inference, Dalmasso et al. [21] use classiﬁers to estimate likelihood ratio statistics and propose a procedurefor guaranteeing valid hypothesis tests and conﬁdence sets.
Finally, Dalmasso et al.
[22] propose apractical procedure for the Neyman construction of conﬁdence sets with ﬁnite-sample guarantees ofnominal coverage as well as diagnostics that estimate conditional coverage over the entire parameterspace.
In this work, we make the assumption that the simulator is well-speciﬁed, in the sense that itaccurately models the real data generation process.
However, this assumption is often violated.
Toovercome this issue, Generalized Bayesian inference (GBI) extends Bayesian inference by replacingthe likelihood term by an arbitrary loss function.
In particular, those loss functions can be designedto mitigate speciﬁc types of misspeciﬁcations and enable robust inference, even with intractablelikelihoods [23–25].
Recently, Dellaporta et al.
[26] further improved GBI by combining it with
Bayesian non-parametric learning, making inference with misspeciﬁed simulator models both robustand computationally efﬁcient.
Conclusions and future work
In this work, we introduced Balanced Neural Ratio Estimation ( BNRE ), a variation of neural ratioestimation designed to produce more conservative posterior estimators, even when the likelihood-toevidence ratio estimator is not faithful.
We provide theoretical arguments that suggest that enforcingthe balancing condition should lead to more conservative posteriors without sacriﬁcing exactnessin the large simulation budget regime.
Our theoretical results are experimentally validated onbenchmarks of varying complexity.
Nevertheless, our inference algorithm comes with limitations that practitioners should keep in mind.
First, we emphasize that theorems 1 and 2 hold only in expectation, which means that we cannotprovide any guarantee at the level of single inferences.
Second, the balancing condition is enforcedthrough a regularization penalty that is not estimated exactly.
This implies that the classiﬁer ˆ dis rarely strictly balanced, although close to be, in which case theorems 1 and 2 do not hold.
Inconclusion, BNRE should not be viewed as a way to obtain conservative posterior estimatorswith 100% reliability, but rather as a way to increase the reliability of the posterior estimatorswith minimal effort and no computational overhead .
Looking forward, the balancing condition could potentially be applied to other simulation-basedinference algorithms.
Future works could include a generalization to neural posterior estimation(NPE).
In fact, the likelihood-to-evidence ratio can be extracted from an approximate posterior byremoving its dependence on the prior, log ˆ r ( x | ϑ ) = log ˆ p ( ϑ | x ) − log p ( ϑ ) , which in turn can beexpressed as a classiﬁer ˆ d ( ϑ , x ) =
σ (log ˆ r ( x | ϑ )) on which the balancing condition can be evaluatedand enforced.
Although our work focuses on amortized approximate inference, the balancingcondition could also be applied to sequential inference algorithms to increase their reliability.
Finally, although our initial motivation is framed within the ﬁeld of simulation-based inference, ourtheoretical results are directly applicable to any binary classiﬁcation task by replacing the joint andmarginal distributions in the balancing condition with the distributions of the two considered classes.
Therefore, it provides an easy-to-implement modiﬁcation for high-risk classiﬁcation problems.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See attached source code orrepository.
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[No] There are none to our knowledge.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes](b) Did you include complete proofs of all theoretical results?
[Yes]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material or as a URL)?
[Yes] See thesupplementary materials in attachment.
(b) Did you specify all the training details (e.g., data splits, hyper-parameters, how theywere chosen)?
[Yes] See section 4 and Appendix D.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
[No]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A] We use none.
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[N/A] We use none.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A] We use none.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]