Original: B RIDGE THE I NFERENCE G APS OF N EURAL P ROCESSES VIA E XPECTATION M AXIMIZATION
Revised: B RIDGE THE I NFERENCE G APS OF N EURAL P ROCESSES VIA E XPECTATION M AXIMIZATION

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The neural process (NP) is a family of computationally efficient models for learning distributions over functions.
Revised: The neural process (NP) is a family of computationally efficient models for learning distributions over functions.

Original: However, it suffers from under-fitting and shows suboptimal performance in practice.
Revised: However, it suffers from under-fitting and shows suboptimal performance in practice.

Original: Researchers have primarily focused on incorporating diverse structural inductive biases, e.g. attention or convolution, in modeling.
Revised: Researchers have primarily focused on incorporating diverse structural inductive biases, e.g. attention or convolution, in modeling.

Original: The topic of inference suboptimality and an analysis of the NP from the optimization objective perspective has hardly been studied in earlier work.
Revised: The topic of inference suboptimality and an analysis of the NP from the optimization objective perspective has hardly been studied in earlier work.

Original: To fix this issue, we propose a surrogate objective of the target log-likelihood of the meta dataset within the expectation maximization framework.
Revised: To fix this issue, we propose a surrogate objective of the target log-likelihood of the meta dataset within the expectation maximization framework.

Original: The resulting model, referred to as the Self-normalized Importance weighted Neural Process (SI-NP), can learn a more accurate functional prior and has an improvement guarantee concerning the target log-likelihood.
Revised: The resulting model, referred to as the Self-normalized Importance weighted Neural Process (SI-NP), can learn a more accurate functional prior and has an improvement guarantee concerning the target log-likelihood.

Original: Experimental results show the competitive performance of SI-NP over other NPs objectives and illustrate that structural inductive biases, such as attention modules, can also augment our method to achieve SOTA performance.
Revised: Experimental results show the competitive performance of SI-NP over other NPs objectives and illustrate that structural inductive biases, such as attention modules, can also augment our method to achieve SOTA performance.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The global latent variable z is to summarize function properties.
Revised: The global latent variable z is to summarize function properties.

Original: The model involves a functional prior distribution p ( z |D C ; ϑ ) and a functional generative distribution p ( D T | z ; ϑ ) .
Revised: The model involves a functional prior distribution p ( z |D C ; ϑ ) and a functional generative distribution p ( D T | z ; ϑ ) .

Original: Please refer to
Revised: Please refer to

Original: Section (2) for detailed notation descriptions.
Revised: Section (2) for detailed notation descriptions.

Original: The combination of deep neural networks and stochastic processes provides a promising framework for modeling data points with correlations (Ghahramani, 2015).
Revised: The combination of deep neural networks and stochastic processes provides a promising framework for modeling data points with correlations (Ghahramani, 2015).

Original: It exploits the high capacity of deep neural networks and enables uncertainty quantification for distributions over functions.
Revised: It exploits the high capacity of deep neural networks and enables uncertainty quantification for distributions over functions.

Original: As an example, we can look at the deep Gaussian process (Damianou & Lawrence, 2013).
Revised: As an example, we can look at the deep Gaussian process (Damianou & Lawrence, 2013).

Original: However, the run-time complexity of predictive distributions in Gaussian processes is cubic w.r.t.
Revised: However, the run-time complexity of predictive distributions in Gaussian processes is cubic w.r.t.

Original: the number of predicted data points.
Revised: the number of predicted data points.

Original: To circumvent this, Garnelo et al.
Revised: To circumvent this, Garnelo et al.

Original: (2018a;b) developed the family of neural processes (NPs) as the alternative, which can model more flexible function distributions and capture predictive uncertainty at a lower computational cost.
Revised: (2018a;b) developed the family of neural processes (NPs) as the alternative, which can model more flexible function distributions and capture predictive uncertainty at a lower computational cost.

Original: In this paper, we study the vanilla NP as a deep latent variable model and show the generative process in Fig.
Revised: In this paper, we study the vanilla NP as a deep latent variable model and show the generative process in Fig.

Original: In particular, let us recap the inference methods used in vanilla NPs: It learns to approximate the functional posterior q ϕ ( z )
Revised: In particular, let us recap the inference methods used in vanilla NPs: It learns to approximate the functional posterior q ϕ ( z )

Original: ≈ p ( z |D T ; ϑ ) and a functional prior q ϕ ( z |D C )
Revised: ≈ p ( z |D T ; ϑ ) and a functional prior q ϕ ( z |D C )

Original: ≈ p ( z |D C ; ϑ ) , which are permutation invariant to the order of data points.
Revised: ≈ p ( z |D C ; ϑ ) , which are permutation invariant to the order of data points.

Original: Then the predictive distribution for a data point [ x ∗ , y ∗ ] can be formulated in the form E q ϕ
Revised: Then the predictive distribution for a data point [ x ∗ , y ∗ ] can be formulated in the form E q ϕ

Original: ( z |D C )
Revised: ( z |D C )

Original: [ p ( y ∗ | [ x ∗ , z ]; ϑ )] .
Revised: [ p ( y ∗ | [ x ∗ , z ]; ϑ )] .

Original: While the NP provides a computationally efficient framework for modeling exchangeable stochastic processes, it exhibits underfitting and fails to capture accurate uncertainty (Garnelo et al., 2018b; Kim et al., 2019) in practice.
Revised: While the NP provides a computationally efficient framework for modeling exchangeable stochastic processes, it exhibits underfitting and fails to capture accurate uncertainty (Garnelo et al., 2018b; Kim et al., 2019) in practice.

Original: To improve its generalization capability, researchers have focused much attention on finding appropriate inductive biases, e.g. attention (Kim et al., 2019) and convolutional modules (Gordon et al., 2019; Kawano et al., 2020), Bayesian mixture structures (Wang & van Hoof, 2022) orBayesian hierarchical structures(Naderiparizi et al., 2020), to incorporate in modeling.
Revised: To improve its generalization capability, researchers have focused much attention on finding appropriate inductive biases, e.g. attention (Kim ∗ Correspondence Author. or structures

Original: Research Motivations.
Revised: Research Motivations.

Original: Most previous work (Garnelo et al., 2018a;b; Kim et al., 2019; Gordon et al., 2019; Wang & van Hoof, 2022) ignores the reason why the vanilla NP suffers the performance bottleneck and what kind of functional priors the vanilla NPs can represent.
Revised: Most previous work (Garnelo et al., 2018a;b; Kim et al., 2019; Gordon et al., 2019; Wang & van Hoof, 2022) ignores the reason why the vanilla NP suffers the performance bottleneck and what kind of functional priors the vanilla NPs can represent.

Original: In particular, we point out the remaining crucial issues that have not been sufficiently investigated in this domain, respectively: (i) understanding the inference suboptimality of vanilla NPs (ii) quantifying statistical traits of learned functional priors.
Revised: In particular, we point out the remaining crucial issues that have not been sufficiently investigated in this domain, respectively: (i) understanding the inference suboptimality of vanilla NPs (ii) quantifying statistical traits of learned functional priors.

Original: To this end, we try to diagnose the vanilla NP from its optimization objective.
Revised: To this end, we try to diagnose the vanilla NP from its optimization objective.

Original: Our primary interest is to find a tractable way to optimize NPs and examine the statistics of learned functional priors from diverse optimization objectives.
Revised: Our primary interest is to find a tractable way to optimize NPs and examine the statistics of learned functional priors from diverse optimization objectives.

Original: Developed Methods.
Revised: Developed Methods.

Original: To understand the inference suboptimality of vanilla NPs, we establish connections among a collection of optimization objectives, e.g. approximate evidence lower bounds (ELBOs) and Monte Carlo estimates of log-likelihoods, in Section (3).
Revised: To understand the inference suboptimality of vanilla NPs, we establish connections among a collection of optimization objectives, e.g. approximate evidence lower bounds (ELBOs) and Monte Carlo estimates of log-likelihoods, in Section (3).

Original: Then we formulate a tractable optimization objective within the variational expectation maximization framework and obtain the Self-normalized Importance weighted neural process (SI-NP) in Section (4).
Revised: Then we formulate a tractable optimization objective within the variational expectation maximization framework and obtain the Self-normalized Importance weighted neural process (SI-NP) in Section (4).

Original: Contributions.
Revised: Contributions.

Original: To summarize, our primary contributions are three-fold: (i) we analyze the inherent inference sub-optimality of NPs from an optimization objective optimization perspective; (ii) we demonstrate the equivalence of conditional NPs (Garnelo et al., 2018a) and SI-NPs with one Monte
Revised: To summarize, our primary contributions are three-fold: (i) we analyze the inherent inference sub-optimality of NPs from an optimization objective optimization perspective; (ii) we demonstrate the equivalence of conditional NPs (Garnelo et al., 2018a) and SI-NPs with one Monte

Original: Carlo sample estimate, which closely relates to the prior collapse in Definition (3.1); (iii) our developed SI-NPs have an improvement guarantee to the likelihood of meta dataset in optimization and show a significant advantage over baselines with other objectives.
Revised: Carlo sample estimate, which closely relates to the prior collapse in Definition (3.1); (iii) our developed SI-NPs have an improvement guarantee to the likelihood of meta dataset in optimization and show a significant advantage over baselines with other objectives.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: General Notations.
Revised: General Notations.

Original: We study NPs in a meta learning setup.
Revised: We study NPs in a meta learning setup.

Original: T defines a set of tasks with τ a sampled task.
Revised: T defines a set of tasks with τ a sampled task.

Original: Let D Cτ = { ( x i , y i ) } ni
Revised: Let D Cτ = { ( x i , y i ) } ni

Original: =1 and D Tτ = { ( x i , y i ) } n + m i =
Revised: =1 and D Tτ = { ( x i , y i ) } n + m i =

Original: 1 denote the context points for the functional prior inference and the target points for the function prediction.
Revised: 1 denote the context points for the functional prior inference and the target points for the function prediction.

Original: The latent variable z is a functional representation of a task τ with observed data points.
Revised: The latent variable z is a functional representation of a task τ with observed data points.

Original: We refer to ϑ ∈ Θ as the parameters of the deep latent variable model for NPs.
Revised: We refer to ϑ ∈ Θ as the parameters of the deep latent variable model for NPs.

Original: In detail, ϑ consists of encoder parameters in a functional prior p ( z |D
Revised: In detail, ϑ consists of encoder parameters in a functional prior p ( z |D

Original: Cτ ; ϑ ) and decoder parameters in a generative distribution p
Revised: Cτ ; ϑ ) and decoder parameters in a generative distribution p

Original: ( D Tτ | z ; ϑ ) .
Revised: ( D Tτ | z ; ϑ ) .

Original: ϕ refer to the parameters of a variational posterior distribution q ϕ ( z ) =
Revised: ϕ refer to the parameters of a variational posterior distribution q ϕ ( z ) =

Original: q ϕ ( z |D Tτ ) , while η refer to the parameters of a proposal distribution q η ( z ) in the following self-normalized importance sampling.
Revised: q ϕ ( z |D Tτ ) , while η refer to the parameters of a proposal distribution q η ( z ) in the following self-normalized importance sampling.

Original: Gaussian distributions with diagonal covariance matrices are the default choice for these distributions, e.g. p ( z |D
Revised: Gaussian distributions with diagonal covariance matrices are the default choice for these distributions, e.g. p ( z |D

Original: Cτ ; ϑ ) =
Revised: Cτ ; ϑ ) =

Original: N ( z ; µ ϑ ( D Cτ ) , Σ ϑ ( D Cτ )) , q ϕ ( z ) =
Revised: N ( z ; µ ϑ ( D Cτ ) , Σ ϑ ( D Cτ )) , q ϕ ( z ) =

Original: N ( z ; µ ϕ ( D Tτ ) , Σ ϕ ( D Tτ )) and q η ( z |D Tτ ) =
Revised: N ( z ; µ ϕ ( D Tτ ) , Σ ϕ ( D Tτ )) and q η ( z |D Tτ ) =

Original: N ( z ; µ η ( D Tτ ) , Σ η ( D Tτ )) .
Revised: N ( z ; µ η ( D Tτ ) , Σ η ( D Tτ )) .

Original: NPs as Exchangeable Stochastic Processes.
Revised: NPs as Exchangeable Stochastic Processes.

Original: In vanilla NPs, the element-wise generative process can be translated into Eq.
Revised: In vanilla NPs, the element-wise generative process can be translated into Eq.

Original: Here the mean and variance functions are respectively denoted by µ and Σ .
Revised: Here the mean and variance functions are respectively denoted by µ and Σ .

Original: [Equation]
Revised: [Equation]

Original: Based on the Kolmogorov extension theorem (Klenke, 2013) and de Finneti’s theorem (Kerns &
Revised: Based on the Kolmogorov extension theorem (Klenke, 2013) and de Finneti’s theorem (Kerns &

Original: Sz´ekely, 2006), the above equation ρ x 1: n + m ( y 1: n + m ) is verified to be a well-defined exchangeable stochastic process.
Revised: Sz´ekely, 2006), the above equation ρ x 1: n + m ( y 1: n + m ) is verified to be a well-defined exchangeable stochastic process.

Original: NPs in Meta Learning Tasks.
Revised: NPs in Meta Learning Tasks.

Original: Given a collection of tasks T , we can decompose the marginal distribution p ( D T T |D C T ; ϑ ) with a global latent variable z in Eq.
Revised: Given a collection of tasks T , we can decompose the marginal distribution p ( D T T |D C T ; ϑ ) with a global latent variable z in Eq.

Original: Here the conditional distribution p ( z |D
Revised: Here the conditional distribution p ( z |D

Original: Cτ ; ϑ ) with τ ∈ T is permutation invariant w.r.t.
Revised: Cτ ; ϑ ) with τ ∈ T is permutation invariant w.r.t.

Original: the order of data points and encodes the functional prior in the generative process.
Revised: the order of data points and encodes the functional prior in the generative process.

Original: [Equation]
Revised: [Equation]

Original: Throughout the paper, the optimization objective of our interest is the marginal log-likelihood of a meta learning dataset in Eq.
Revised: Throughout the paper, the optimization objective of our interest is the marginal log-likelihood of a meta learning dataset in Eq.

Original: Furthermore, this applies to all NP variants.
Revised: Furthermore, this applies to all NP variants.

Original: [Equation]
Revised: [Equation]

Original: For the sake of simplicity, we consider one task τ to derive equations in the following section, which corresponds to maximizing the following objective 1 .
Revised: For the sake of simplicity, we consider one task τ to derive equations in the following section, which corresponds to maximizing the following objective 1 .

Original: [Equation]
Revised: [Equation]

Original: In the NP family (Garnelo et al., 2018a;b), the target data points are conditionally independent given the global latent variable p ( D Tτ | z ; ϑ )
Revised: In the NP family (Garnelo et al., 2018a;b), the target data points are conditionally independent given the global latent variable p ( D Tτ | z ; ϑ )

Original: = (cid:81) n + m i =1 p
Revised: = (cid:81) n + m i =1 p

Original: y i |
Revised: y i |

Original: [ x i , z ]; ϑ ) .
Revised: [ x i , z ]; ϑ ) .

Original: The marginal distribution can be interpreted as the infinite mixture of distributions when z is defined on a continuous domain.
Revised: The marginal distribution can be interpreted as the infinite mixture of distributions when z is defined on a continuous domain.

Original: Now learning distributions over functions is reduced to a probabilistic inference problem.
Revised: Now learning distributions over functions is reduced to a probabilistic inference problem.

Original:  O PTIMIZATION G APS AND S TATISTICAL T RAITS 3. I NFERENCE S UBOPTIMALITY IN VANILLA NP S
Revised:  O PTIMIZATION G APS AND S TATISTICAL T RAITS 3. I NFERENCE S UBOPTIMALITY IN VANILLA NP S

Original: Previously, variational auto-encoder (VAE) models (Kingma & Welling, 2013; Rezende et al., 2014) mostly set a prior distribution fixed, e.g. N (0 , I ) , as the default to approximate the posterior.
Revised: Previously, variational auto-encoder (VAE) models (Kingma & Welling, 2013; Rezende et al., 2014) mostly set a prior distribution fixed, e.g. N (0 , I ) , as the default to approximate the posterior.

Original: This differs significantly from NPs family settings.
Revised: This differs significantly from NPs family settings.

Original: On the one hand, the functional prior is learned in
Revised: On the one hand, the functional prior is learned in

Original: NPs.
Revised: NPs.

Original: On the other hand, the functional prior participates in the performance evaluation.
Revised: On the other hand, the functional prior participates in the performance evaluation.

Original: Exact ELBO for NPs.
Revised: Exact ELBO for NPs.

Original: Following the essential variational inference operation, we can establish connections between the exact ELBO and the log-likelihood in Eq.
Revised: Following the essential variational inference operation, we can establish connections between the exact ELBO and the log-likelihood in Eq.

Original: Given the functional prior p ( z |D
Revised: Given the functional prior p ( z |D

Original: Cτ ; ϑ ) and the generative distribution p ( D Tτ | z ; ϑ ) , the exact functional posterior can be obtained by the Bayes rule
Revised: Cτ ; ϑ ) and the generative distribution p ( D Tτ | z ; ϑ ) , the exact functional posterior can be obtained by the Bayes rule

Original: [Equation]
Revised: [Equation]

Original: The denominator p ( D Tτ | D Cτ ) makes exact inference infeasible, and the family of variational posteriors is introduced to approximate p ( z |D Tτ ; ϑ ) .
Revised: The denominator p ( D Tτ | D Cτ ) makes exact inference infeasible, and the family of variational posteriors is introduced to approximate p ( z |D Tτ ; ϑ ) .

Original: Here the variational posterior family is defined in a parameterized set Q Φ = { q ϕ ( z ) | ϕ ∈ Φ } .
Revised: Here the variational posterior family is defined in a parameterized set Q Φ = { q ϕ ( z ) | ϕ ∈ Φ } .

Original: [Equation]
Revised: [Equation]

Original: When the variational posterior family is flexible enough, e.g. p ( z |D Tτ ; ϑ ) ∈ Q Φ , the posterior approximation gap can be reduced to an arbitrarily small quantity.
Revised: When the variational posterior family is flexible enough, e.g. p ( z |D Tτ ; ϑ ) ∈ Q Φ , the posterior approximation gap can be reduced to an arbitrarily small quantity.

Original: In this case, maximizing the exact ELBO in Eq.
Revised: In this case, maximizing the exact ELBO in Eq.

Original: (6) increases the likelihood in Eq.
Revised: (6) increases the likelihood in Eq.

Original: (5) accordingly.
Revised: (5) accordingly.

Original: [Equation]
Revised: [Equation]

Original: Approximate ELBO for NPs.
Revised: Approximate ELBO for NPs.

Original: As previously mentioned, the inference is complicated since the functional prior and the posterior in the exact ELBO are unknown.
Revised: As previously mentioned, the inference is complicated since the functional prior and the posterior in the exact ELBO are unknown.

Original: To this end, Garnelo et al.
Revised: To this end, Garnelo et al.

Original: (2018b) proposes a surrogate objective as an approximate ELBO for NPs.
Revised: (2018b) proposes a surrogate objective as an approximate ELBO for NPs.

Original: This is defined as Eq.
Revised: This is defined as Eq.

Original: (7) to maximize.
Revised: (7) to maximize.

Original: [Equation]
Revised: [Equation]

Original: The Kullback-Leibler divergence between the approximate posterior q ϕ ( z ) and the approximate prior q ϕ ( z |D Cτ ) is referred to as the consistent regularizer in this paper.
Revised: The Kullback-Leibler divergence between the approximate posterior q ϕ ( z ) and the approximate prior q ϕ ( z |D Cτ ) is referred to as the consistent regularizer in this paper.

Original: We claim that the consistent regularizer is the source of the inference suboptimality of vanilla NPs, and this is shown in Appendix (D.2) as the proof of Remark (1).
Revised: We claim that the consistent regularizer is the source of the inference suboptimality of vanilla NPs, and this is shown in Appendix (D.2) as the proof of Remark (1).

Original: Remark 1 Eq.
Revised: Remark 1 Eq.

Original: (7) is an invalid variational inference objective, and optimizing it cannot guarantee to find optimal or locally optimal solutions for the maximization over (cid:80)
Revised: (7) is an invalid variational inference objective, and optimizing it cannot guarantee to find optimal or locally optimal solutions for the maximization over (cid:80)

Original: τ
Revised: τ

Original: ∈T ln p ( D Tτ |D
Revised: ∈T ln p ( D Tτ |D

Original: Cτ ; ϑ ) .
Revised: Cτ ; ϑ ) .

Original: Other Available Objectives in NPs Family.
Revised: Other Available Objectives in NPs Family.

Original: Now we turn to other tractable optimization objectives in NPs.
Revised: Now we turn to other tractable optimization objectives in NPs.

Original: These include conditional neural processes (CNPs) (Garnelo et al., 2018a) and convolutional neural processes (ConvNPs) (Foong et al., 2020) (or VERSA (Gordon et al., 2018)).
Revised: These include conditional neural processes (CNPs) (Garnelo et al., 2018a) and convolutional neural processes (ConvNPs) (Foong et al., 2020) (or VERSA (Gordon et al., 2018)).

Original: The CNP is also a typical model of the NPs family.
Revised: The CNP is also a typical model of the NPs family.

Original: The objective L CNP ( ϑ ) can be obtained when the functional prior collapses into a Dirac delta distribution with ˆ z a fixed real-value vector.
Revised: The objective L CNP ( ϑ ) can be obtained when the functional prior collapses into a Dirac delta distribution with ˆ z a fixed real-value vector.

Original: [Equation]
Revised: [Equation]

Original: For the ConvNP, we do not focus on the convolutional structural inductive bias and concentrate more on the optimization objective itself.
Revised: For the ConvNP, we do not focus on the convolutional structural inductive bias and concentrate more on the optimization objective itself.

Original: Its objective in Eq.
Revised: Its objective in Eq.

Original: (9) is a biased Monte Carlo estimate of Eq.
Revised: (9) is a biased Monte Carlo estimate of Eq.

Original: (3), so we can maximize the log-likelihood of marginal distributions straightforwardly.
Revised: (3), so we can maximize the log-likelihood of marginal distributions straightforwardly.

Original: [Equation]
Revised: [Equation]

Original: This is termed as Monte Carlo Maximum Likelihood L ML-NP ( ϑ ) 2 , and B is the number of used
Revised: This is termed as Monte Carlo Maximum Likelihood L ML-NP ( ϑ ) 2 , and B is the number of used

Original: Monte Carlo samples.
Revised: Monte Carlo samples.

Original: Without the involvement of the consistent regularizer, both L CNP ( ϑ ) and
Revised: Without the involvement of the consistent regularizer, both L CNP ( ϑ ) and

Original: L ML-NP ( ϑ ) will not encounter the approximation gap in practice.
Revised: L ML-NP ( ϑ ) will not encounter the approximation gap in practice.

Original: 3. E VALUATION C RITERIA & A SYMPTOTIC P ERFORMANCE
Revised: 3. E VALUATION C RITERIA & A SYMPTOTIC P ERFORMANCE

Original: As in (Le et al., 2018; Foong et al., 2020), we take a multi-sample Monte Carlo method to evaluate the performance.
Revised: As in (Le et al., 2018; Foong et al., 2020), we take a multi-sample Monte Carlo method to evaluate the performance.

Original: This applies to both meta training and meta testing processes.
Revised: This applies to both meta training and meta testing processes.

Original: In detail, NP models need to run B times stochastic forward pass by sampling z ( b ) ∼ p ( z |D
Revised: In detail, NP models need to run B times stochastic forward pass by sampling z ( b ) ∼ p ( z |D

Original: Cτ ; ϑ ) and then compute the log-likelihoods as ln (cid:104) 1 B (cid:80)
Revised: Cτ ; ϑ ) and then compute the log-likelihoods as ln (cid:104) 1 B (cid:80)

Original: Bb =1 p ( D Tτ | z ( b ) ; ϑ ) (cid:105) .
Revised: Bb =1 p ( D Tτ | z ( b ) ; ϑ ) (cid:105) .

Original: After describing the evaluation criteria, we turn to a phenomenon of our interest.
Revised: After describing the evaluation criteria, we turn to a phenomenon of our interest.

Original: In Gaussian processes (Ghahramani, 2015), with more observed context points, the epistemic uncertainty can be decreased, and the predictive mean function is closer to the ground truth.
Revised: In Gaussian processes (Ghahramani, 2015), with more observed context points, the epistemic uncertainty can be decreased, and the predictive mean function is closer to the ground truth.

Original: Similarly, this trait is also reflected in NPs family and can be quantitatively described as follows.
Revised: Similarly, this trait is also reflected in NPs family and can be quantitatively described as follows.

Original: Given a measure of average predictive errors β , the number of context points n and the evaluated dataset, D Tτ , the introduced metrics β
Revised: Given a measure of average predictive errors β , the number of context points n and the evaluated dataset, D Tτ , the introduced metrics β

Original: ( D Tτ ; n ) are decreased when increasing n in prediction.
Revised: ( D Tτ ; n ) are decreased when increasing n in prediction.

Original: In our paper, we refer to this trait as the asymptotic behavior .
Revised: In our paper, we refer to this trait as the asymptotic behavior .

Original: Definition 3.1 (Prior Collapse)
Revised: Definition 3.1 (Prior Collapse)

Original: The functional prior p ( z |D
Revised: The functional prior p ( z |D

Original: Cτ ; ϑ ) =
Revised: Cτ ; ϑ ) =

Original: N ( z ; µ ϑ ( D Cτ ) , Σ ϑ ( D Cτ )) is said to collapse in learning when the trace of the covariance matrix satisfies
Revised: N ( z ; µ ϑ ( D Cτ ) , Σ ϑ ( D Cτ )) is said to collapse in learning when the trace of the covariance matrix satisfies

Original: Tr [Σ ϑ ( D Cτ )] = (cid:80) di =1 σ 2 i ≈ 0 with Σ
Revised: Tr [Σ ϑ ( D Cτ )] = (cid:80) d i =1 σ 2 i ≈ 0 with Σ

Original: ϑ ( D Cτ ) =
Revised: ϑ ( D C τ ) =

Original: diag [ σ 21 , . . 
Revised: diag [ σ 2 1 , . . .

Original: , σ 2 d ] .
Revised: , σ 2 d ] .

Original: As previously mentioned, we can more precisely keep track of measures β
Revised: As previously mentioned, we can more precisely keep track of measures β

Original: ( D Tτ ; n ) , such as predictive log-likelihoods or mean square errors of data points, to assess the asymptotic behavior.
Revised: ( D Tτ ; n ) , such as predictive log-likelihoods or mean square errors of data points, to assess the asymptotic behavior.

Original: The role of latent variables z is to propagate the uncertainty about the partial observations in functions.
Revised: The role of latent variables z is to propagate the uncertainty about the partial observations in functions.

Original: And
Revised: And

Original: Definition (3.1) provides a quantitative way to examine the extent of prior collapse in ML-NPs and
Revised: Definition (3.1) provides a quantitative way to examine the extent of prior collapse in ML-NPs and

Original: SI-NPs.
Revised: SI-NPs.

Original:  T RACTABLE O PTIMIZATION VIA E XPECTATION M AXIMIZATION
Revised:  T RACTABLE O PTIMIZATION VIA E XPECTATION M AXIMIZATION

Original: In this section, we propose alleviating the inference suboptimality of NPs with the help of the variational expectation maximization algorithm.
Revised: In this section, we propose alleviating the inference suboptimality of NPs with the help of the variational expectation maximization algorithm.

Original: The strategy is to formulate a surrogate optimization objective and then execute the EM -steps in optimization.
Revised: The strategy is to formulate a surrogate optimization objective and then execute the EM -steps in optimization.

Original: The benefit of our method is to guarantee performance improvement w.r.t.
Revised: The benefit of our method is to guarantee performance improvement w.r.t.

Original: the likelihood of meta dataset in iterations and finally result in at least a local optimum.
Revised: the likelihood of meta dataset in iterations and finally result in at least a local optimum.

Revised: [Figure]

Original: 4. V ARIATIONAL E XPECTATION M AXIMIZATION FOR NP S
Revised: 4. V ARIATIONAL E XPECTATION M AXIMIZATION FOR NP S

Original: This part is to avoid the inference suboptimality of vanilla NPs as mentioned earlier.
Revised: This part is to avoid the inference suboptimality of vanilla NPs as mentioned earlier.

Original: We retain the neural architectures used in NPs.
Revised: We retain the neural architectures used in NPs.

Original: The basic idea is illustrated in Fig.
Revised: The basic idea is illustrated in Fig.

Original: In detail, we iteratively 
Revised: In detail, we iteratively construct the lower bound L ( ϑ K ) and maximize the surrogate function L ( ϑ ; ϑ K ) . The referred optimization gap is due to the complexity of objectives or the choice of optimizers and measures the difference converged optimal functional prior the theoretical optimal functional Algorithm Input : Task

Original: [Figure]
Revised: Initialized

Original: Algorithm 1: Variational Expectation Maximization for NPs.
Revised: Output: Meta-trained parameters ϑ and η . 1 for k = 1 to K do ; if Use the Functional Prior as the Proposal then ; else

Original:  S URROGATE F UNCTION FOR E XACT NP S
Revised: 4.1. S URROGATE F UNCTION FOR E XACT NP S

Original: To make the optimization of meta dataset log-likelihood feasible, we construct surrogate functions as the proxy in each iteration step.
Revised: To make the optimization of meta dataset log-likelihood feasible, we construct surrogate functions as the proxy in each iteration step.

Original: These meta learning surrogate functions with special properties are closely connected with the original objective, and we leave this discussion in Appendix (E).
Revised: These meta learning surrogate functions with special properties are closely connected with the original objective, and we leave this discussion in Appendix (E).

Original: Here ϑ k denotes the parameter of the latent variable model for NPs in the k -th iteration of variational expectation maximization.
Revised: Here ϑ k denotes the parameter of the latent variable model for NPs in the k -th iteration of variational expectation maximization.

Original: Following the Algorithm (1), we take the E -step #1 by replacing the approximate posterior in Eq.
Revised: Following the Algorithm (1), we take the E -step #1 by replacing the approximate posterior in Eq.

Original: (6) with the last time updated p ( z |D Tτ ; ϑ k ) .
Revised: (6) with the last time updated p ( z |D Tτ ; ϑ k ) .

Original: And this results in the following equation,
Revised: And this results in the following equation,

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where p ( z |D Tτ ; ϑ k ) is the posterior distribution.
Revised: where p ( z |D Tτ ; ϑ k ) is the posterior distribution.

Original: Proposition 

Revised: Proposition

Original: The proposed meta learning function L ( ϑ ; ϑ k ) in Eq.
Revised: The proposed meta learning function L ( ϑ ; ϑ k ) in Eq.

Original: (10) is a surrogate function w.r.t.
Revised: (10) is a surrogate function w.r.t.

Original: the log-likelihood of the meta learning dataset.
Revised: the log-likelihood of the meta learning dataset.

Original: The above proposition is examined based on the definition in Appendix (E.1).
Revised: The above proposition is examined based on the definition in Appendix (E.1).

Original: 4.1. T RACTABLE O PTIMIZATION WITH S ELF -N ORMALIZED I MPORTANCE S AMPLING
Revised: 4.1. T RACTABLE O PTIMIZATION WITH S ELF -N ORMALIZED I MPORTANCE S AMPLING

Original: Since the second term in Eq.
Revised: Since the second term in Eq.

Original: (10) is constant in the iteration, we can drop it to simplify the surrogate objective as the right side of the following equation.
Revised: (10) is constant in the iteration, we can drop it to simplify the surrogate objective as the right side of the following equation.

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Proposition 2 Optimizing this surrogate function of a batch of tasks via the variational expectation maximization leads to an improvement guarantee w.r.t.
Revised: Proposition 2 Optimizing this surrogate function of a batch of tasks via the variational expectation maximization leads to an improvement guarantee w.r.t.

Original: the log-likelihood (cid:80) τ
Revised: the log-likelihood (cid:80) τ

Original: ∈T ln p ( D Tτ |D
Revised: ∈T ln p ( D Tτ |D

Original: Cτ ; ϑ ) .
Revised: Cτ ; ϑ ) .

Original: Still we cannot optimize L EM ( ϑ ; ϑ k ) since the expectation has no analytical solution and it is intractable to sample from p ( z |D Tτ ; ϑ k ) for Monte Carlo estimates 3 .
Revised: Still we cannot optimize L EM ( ϑ ; ϑ k ) since the expectation has no analytical solution and it is intractable to sample from p ( z |D Tτ ; ϑ k ) for Monte Carlo estimates 3 .

Original: Remember that the marginal distribution p ( D Tτ |D
Revised: Remember that the marginal distribution p ( D Tτ |D

Original: Cτ ; ϑ k ) is task dependent and can not be ignored in computing the posterior p ( z |D Tτ ; ϑ k ) .
Revised: Cτ ; ϑ k ) is task dependent and can not be ignored in computing the posterior p ( z |D Tτ ; ϑ k ) .

Original: To circumvent this, we introduce a proposal distribution q η ( z |D Tτ ) and optimize the objective via self-normalized importance sampling (Tokdar & Kass, 2010).
Revised: To circumvent this, we introduce a proposal distribution q η ( z |D Tτ ) and optimize the objective via self-normalized importance sampling (Tokdar & Kass, 2010).

Original: The resulting meta learning surrogate function is as follows:
Revised: The resulting meta learning surrogate function is as follows:

Original: [Equation]
Revised: [Equation]

Original: where z ( b ) ∼ q η k ( z |D Tτ ) , ω ( b ) =
Revised: where z ( b ) ∼ q η k ( z |D Tτ ) , ω ( b ) =

Original: exp (cid:0) ln 
Revised: exp (cid:0) ln p

Original: [Equation] and ˆ ω ( b ) = 
Revised: ( D Tτ | z ( b ) ; ϑ k ) + ln p ( z ( b ) |D Cτ ; ϑ k ) − ln q η k ( z ( b ) |D Tτ ) (cid:1) and ˆ ω ( b ) = ω

Original: [Equation]
Revised: [Equation]

Original: In terms of the first conditional term in Eq.
Revised: In terms of the first conditional term in Eq.

Original: (12), all the data points are conditional independent and this is further expressed as ln p ( D Tτ | z ( b ) ; ϑ ) = (cid:80) n + m i =
Revised: (12), all the data points are conditional independent and this is further expressed as ln p ( D Tτ | z ( b ) ; ϑ ) = (cid:80) n + m i =

Original: 1 ln p ( y i |
Revised: 1 ln p ( y i |

Original: [ x i , z ( b ) ]; ϑ ) .
Revised: [ x i , z ( b ) ]; ϑ ) .

Original: In practice, the selection of proposal distributions is empirically tricky, so we make the update of proposal distributions optional in implementations.
Revised: In practice, the selection of proposal distributions is empirically tricky, so we make the update of proposal distributions optional in implementations.

Original: In our experimental settings, we simply use the functional prior p ( z |D
Revised: In our experimental settings, we simply use the functional prior p ( z |D

Original: Cτ ; ϑ ) as the default proposal distribution, which is competitive enough in performance.
Revised: Cτ ; ϑ ) as the default proposal distribution, which is competitive enough in performance.

Original: Proposition 3 With one Monte Carlo sample used in Eq.
Revised: Proposition 3 With one Monte Carlo sample used in Eq.

Original: (12), the presumed diagonal Gaussian prior p ( z |D
Revised: (12), the presumed diagonal Gaussian prior p ( z |D

Original: Cτ ; ϑ ) will collapse into a Dirac delta distribution in convergence.
Revised: Cτ ; ϑ ) will collapse into a Dirac delta distribution in convergence.

Original: In this case, SI-NP with the one sample Monte Carlo estimate in Eq.
Revised: In this case, SI-NP with the one sample Monte Carlo estimate in Eq.

Original: (13) is equivalent with CNP in Eq.
Revised: (13) is equivalent with CNP in Eq.

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The Proposition (3) establishes connections between SI-NPs and CNPs in optimization and we attribute this to the collapse term in Eq.
Revised: The Proposition (3) establishes connections between SI-NPs and CNPs in optimization and we attribute this to the collapse term in Eq.

Original: Hence, CNP can be viewed as a particular example in
Revised: Hence, CNP can be viewed as a particular example in

Original: SI-NPs.
Revised: SI-NPs.

Original: The complete proof is based on limit analysis and can be found in Appendix (F.1).
Revised: The complete proof is based on limit analysis and can be found in Appendix (F.1).

Original: 4. S CALABLE T RAINING AND T ESTING
Revised: 4. S CALABLE T RAINING AND T ESTING

Original: As shown in Algorithm (1), the meta training process consists of two steps.
Revised: As shown in Algorithm (1), the meta training process consists of two steps.

Original: We skip E -step #2 to avoid unstable optimization observed in empirical results.
Revised: We skip E -step #2 to avoid unstable optimization observed in empirical results.

Original: By repeating E -step #1/ M -step iterations until convergence, the method can theoretically find at least a local optimal w.r.t.
Revised: By repeating E -step #1/ M -step iterations until convergence, the method can theoretically find at least a local optimal w.r.t.

Original: the log-likelihood of meta learning dataset based on the Proposition (2).
Revised: the log-likelihood of meta learning dataset based on the Proposition (2).

Original: Once the learning progress reaches the final convergence, we can make use of the learned functional prior to obtain the predictive distribution.
Revised: Once the learning progress reaches the final convergence, we can make use of the learned functional prior to obtain the predictive distribution.

Original: With B particles in prediction, the distribution can be expressed as follows.
Revised: With B particles in prediction, the distribution can be expressed as follows.

Original: [Equation]
Revised: [Equation]

Original:  E XPERIMENTS AND A NALYSIS
Revised:  E XPERIMENTS AND A NALYSIS

Original: In this section, two central questions are answered: (i) can variational EM based models SI-NPs, achieve a better local optimum than vanilla NPs?
Revised: In this section, two central questions are answered: (i) can variational EM based models SI-NPs, achieve a better local optimum than vanilla NPs?

Original: (ii) what is the role of randomness in functional priors?
Revised: (ii) what is the role of randomness in functional priors?

Original: Specifically, we examine the influence of NPs optimization objectives on typical downstream tasks and understand the functional prior quantitatively.
Revised: Specifically, we examine the influence of NPs optimization objectives on typical downstream tasks and understand the functional prior quantitatively.

Original: Baselines & Evaluations.
Revised: Baselines & Evaluations.

Original: Since our concentration is on optimization objectives in NPs family, we compare to NP (Garnelo et al., 2018b), and CNP (Garnelo et al., 2018a), ML-NP (Foong et al., 2020) in experiments.
Revised: Since our concentration is on optimization objectives in NPs family, we compare to NP (Garnelo et al., 2018b), and CNP (Garnelo et al., 2018a), ML-NP (Foong et al., 2020) in experiments.

Original: Note that our developed SI-NP and ML-NP (Foong et al., 2020) are importance weighted models, but the mechanisms in estimating weights are significantly different.
Revised: Note that our developed SI-NP and ML-NP (Foong et al., 2020) are importance weighted models, but the mechanisms in estimating weights are significantly different.

Original: As for evaluations, we refer the reader to Sec.
Revised: As for evaluations, we refer the reader to Sec.

Original: (3.2) for more information.
Revised: (3.2) for more information.

Original: Due to the page limit, additional experimental results are attached in Appendix (H.5), which shows that SI-NPs can achieve SOTA performance by adding attention networks.
Revised: Due to the page limit, additional experimental results are attached in Appendix (H.5), which shows that SI-NPs can achieve SOTA performance by adding attention networks.

Original: 5. S YNTHETIC R EGRESSION
Revised: 5. S YNTHETIC R EGRESSION

Original: [Figure]
Revised: [Figure]

Original: We create synthetic regression tasks by sampling functions from Gaussian processes.
Revised: We create synthetic regression tasks by sampling functions from Gaussian processes.

Original: Three types of kernels, respectively Matern − 52 , RBF, and Periodic, are used to generate diverse function distributions.
Revised: Three types of kernels, respectively Matern − 52 , RBF, and Periodic, are used to generate diverse function distributions.

Original: In each iteration, a batch of data points from functions is randomly processed into the context and the target for training models.
Revised: In each iteration, a batch of data points from functions is randomly processed into the context and the target for training models.

Original: In meta testing, the same strategy is used to generate functions and data points.
Revised: In meta testing [Table] the same strategy is used to generate functions and data points.

Original: We report the average log-likelihood results in Table (1).
Revised: We report the average log-likelihood results in Table (1).

Original: It shows SI-NPs outperform the vanilla NPs in all kernel cases and are at least competitive with other baselines.
Revised: It shows SI-NPs outperform the vanilla NPs in all kernel cases and are at least competitive with other baselines.

Original: ML-NPs are also superior to NPs, but they cannot beat CNPs in RBF and Periodic cases.
Revised: ML-NPs are also superior to NPs, but they cannot beat CNPs in RBF and Periodic cases.

Original: An illustration of fitted curves is given in Fig.
Revised: An illustration of fitted curves is given in Fig.

Original: (3): compared to vanilla NPs, we notice that SI-NPs and ML-NPs can better match the ground truth and capture the uncertainty between context data points.
Revised: (3): compared to vanilla NPs, we notice that SI-NPs and ML-NPs can better match the ground truth and capture the uncertainty between context data points.

Original: [Table]
Revised: [Table]

Original: 5. I MAGE C OMPLETION
Revised: 5. I MAGE C OMPLETION

Original: Similar to experiments in (Garnelo et al., 2018b; Kim et al., 2019), we perform image completion experiments in this section.
Revised: Similar to experiments in (Garnelo et al., 2018b; Kim et al., 2019) [Table] we perform image completion experiments in this section.

Original: We implement NPs baselines in four commonly used datasets, namely
Revised: We implement NPs baselines in four commonly used datasets, namely

Original: MNIST (Bottou et al., 1994), FMNIST (Xiao et al., 2017), CIFAR10 (Krizhevsky et al., 2009) and
Revised: MNIST (Bottou et al., 1994), FMNIST (Xiao et al., 2017), CIFAR10 (Krizhevsky et al., 2009) and

Original: SVHN (Sermanet et al., 2012).
Revised: SVHN (Sermanet et al., 2012).

Original: During the meta training, the goal is to complete images in which some pixels have been randomly masked out.
Revised: During the meta training, the goal is to complete images in which some pixels have been randomly masked out.

Original: More precisely, we learn a latent variable model that maps all pixel locations x ∈ [0 , 1] 2 to Gaussian distribution parameters of pixel values based on the context pixel locations and values.
Revised: More precisely, we learn a latent variable model that maps all pixel locations x ∈ [0 , 1] 2 to Gaussian distribution parameters of pixel values based on the context pixel locations and values.

Original: Fig.
Revised: Fig.

Original: (4) is an example of completion results from sampled images.
Revised: (4) is an example of completion results from sampled images.

Original: For implementation details, please refer to Appendix (G).
Revised: For implementation details, please refer to Appendix (G).

Original: [Table]
Revised: [Table]

Original: In the evaluation, the number of context pixels is randomly selected for each image in the dataset.
Revised: In the evaluation, the number of context pixels is randomly selected for each image in the dataset.

Original: We examine the performance in a testing set of the above image datasets and report the average log-likelihoods in Table (2).
Revised: We examine the performance in a testing set of the above image datasets and report the average log-likelihoods in Table (2).

Original: We can see that SINP achieves the best performance in all image datasets.
Revised: We can see that SINP achieves the best performance in all image datasets.

Original: The performance gaps between SI-NPs and NPs are remarkable.
Revised: The performance gaps between SI-NPs and NPs are remarkable.

Original: Furthermore, the asymptotic behaviors of all baselines are illustrated in Fig.
Revised: Furthermore, the asymptotic behaviors of all baselines are illustrated in Fig.

Original: All models exhibit asymptotic behaviors by varying the number of context points, and NPs mainly result in clearly lower log-likelihoods with 10 context points.
Revised: All models exhibit asymptotic behaviors by varying the number of context points, and NPs mainly result in clearly lower log-likelihoods with 10 context points.

Original: The observations with different number of context pixels are consistent with the conclusion in Table (2).
Revised: The observations with different number of context pixels are consistent with the conclusion in Table (2).

Original: Another critical finding is reported in Fig.
Revised: Another critical finding is reported in Fig.

Original: (6), which examines whether the learned functional priors collapse into the deterministic ones.
Revised: (6), which examines whether the learned functional priors collapse into the deterministic ones.

Original: This is based on the average computed trace of covariance matrices in learned functional priors.
Revised: This is based on the average computed trace of covariance matrices in learned functional priors.

Original: In MNIST dataset, SI-NPs encounter the prior collapse, and
Revised: In MNIST dataset, SI-NPs encounter the prior collapse, and

Original: ML-NPs also obtain extremely lower trace values of covariance matrices.
Revised: ML-NPs also obtain extremely lower trace values of covariance matrices.

Original: We attribute the prior collapse in MNIST to its most superficial structures and limited semantic diversity, which results in lower prior uncertainty.
Revised: We attribute the prior collapse in MNIST to its most superficial structures and limited semantic diversity, which results in lower prior uncertainty.

Original: For image datasets with more complicated semantics, the computed trace values in SI-NPs retain a reasonable interval.
Revised: For image datasets with more complicated semantics, the computed trace values in SI-NPs retain a reasonable interval.

Original: The scale of SI-NPs’ trace values coincides with the semantics complexity of image datasets: CIFAR10 > SVHN > FMNIST > MNIST.
Revised: The scale of SI-NPs’ trace values coincides with the semantics complexity of image datasets: CIFAR10 > SVHN > FMNIST > MNIST.

Original: Another common observation is that with context points increasing, both models’ trace values of functional priors in most datasets decrease to a certain level.
Revised: Another common observation is that with context points increasing, both models’ trace values of functional priors in most datasets decrease to a certain level.

Original: The above reveals the meaning of the learned functional priors in SI-NPs: The functional prior exhibits higher uncertainty with more complicated semantics.
Revised: The above reveals the meaning of the learned functional priors in SI-NPs: The functional prior exhibits higher uncertainty with more complicated semantics.

Original: Hence, it has the potential of generating functional samples with more diversity.
Revised: Hence, it has the potential of generating functional samples with more diversity.

Original: For datasets with less rich semantics, the randomness of the learned functional priors plays a less critical role, and the deterministic functional representation is capable of function generation.
Revised: For datasets with less rich semantics, the randomness of the learned functional priors plays a less critical role, and the deterministic functional representation is capable of function generation.

Original: Also, with more observations, the randomness of functional priors can be reduced.
Revised: Also, with more observations, the randomness of functional priors can be reduced.

Original: [Figure]
Revised: [Figure]

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: NPs Family.
Revised: NPs Family.

Original: NPs are simple and flexible in model formulations, but they may suffer from underfitting.
Revised: NPs are simple and flexible in model formulations, but they may suffer from underfitting.

Original: Previous research focuses more on the use of structural inductive biases in NPs.
Revised: Previous research focuses more on the use of structural inductive biases in NPs.

Original: (Kim et al., 2019; 2021) improves the predictive performance by adding attention based local variables.
Revised: (Kim et al., 2019; 2021) improves the predictive performance by adding attention based local variables.

Original: Translation equivariance and invariance are incorporated in modeling NPs with help of convolutions (Gordon et al., 2019;
Revised: Translation equivariance and invariance are incorporated in modeling NPs with help of convolutions (Gordon et al., 2019;

Original: Foong et al., 2020; Kawano et al., 2020).
Revised: Foong et al., 2020; Kawano et al., 2020).

Original: To find more compact functional representations, the contrastive loss is used to regularize (C)NPs’ training (Gondal et al., 2021).
Revised: To find more compact functional representations, the contrastive loss is used to regularize (C)NPs’ training (Gondal et al., 2021).

Original: Hierarchical and mixture structures are also explored to induce diverse latent variables in NPs (Wang & Van Hoof, 2020; Wang & van Hoof, 2022).
Revised: Hierarchical and mixture structures are also explored to induce diverse latent variables in NPs (Wang & Van Hoof, 2020; Wang & van Hoof, 2022).

Original: Lee et al.
Revised: Lee et al.

Original: (2020) combines boostrapping tricks with neural processes to improve expressiveness of latent variables.
Revised: (2020) combines boostrapping tricks with neural processes to improve expressiveness of latent variables.

Original: Besides, NPs are applied to address sequential decision-making problems (Galashov et al., 2019; Wang & Van Hoof, 2022).
Revised: Besides, NPs are applied to address sequential decision-making problems (Galashov et al., 2019; Wang & Van Hoof, 2022).

Original: A summary of these models is given in Appendix (D.3).
Revised: A summary of these models is given in Appendix (D.3).

Original: Expectation Maximization & Wake-Sleep Algorithms.
Revised: Expectation Maximization & Wake-Sleep Algorithms.

Original: For log-likelihood maximization problems with incomplete observations, expectation maximization (Bishop & Nasrabadi, 2006;
Revised: For log-likelihood maximization problems with incomplete observations, expectation maximization (Bishop & Nasrabadi, 2006;

Original: Balestriero et al., 2020) is a tractable approach in optimization.
Revised: Balestriero et al., 2020) is a tractable approach in optimization.

Original: It consists of an E -step to optimize the distribution of latent variables and a M -step to maximize the likelihood parameters.
Revised: It consists of an E -step to optimize the distribution of latent variables and a M -step to maximize the likelihood parameters.

Original: The optimization of VAEs and variants is also built upon an EM framework (Ghojogh et al., 2021; Gao et al., 2021).
Revised: The optimization of VAEs and variants is also built upon an EM framework (Ghojogh et al., 2021; Gao et al., 2021).

Original: Our developed algorithm can be interpreted as EM for NPs.
Revised: Our developed algorithm can be interpreted as EM for NPs.

Original: Another family of algorithms related to our method is wake-sleep algorithms (Bornschein & Bengio, 2014; Eslami et al., 2016; Dieng & Paisley, 2019; Le et al., 2020), where self-normalized importance sampling is used in Monte Carlo estimates.
Revised: Another family of algorithms related to our method is wake-sleep algorithms (Bornschein & Bengio, 2014; Eslami et al., 2016; Dieng & Paisley, 2019; Le et al., 2020), where self-normalized importance sampling is used in Monte Carlo estimates.

Original: In this case, our method can be also viewed as the extension of the reweighted wake-sleep (RWS) algorithm to NPs with an improvement guarantee.
Revised: In this case, our method can be also viewed as the extension of the reweighted wake-sleep (RWS) algorithm to NPs with an improvement guarantee.

Original: Another difference lies in that optimizing a learnable functional prior is of most importance in NPs, while RWS algorithms are mostly used in scenarios with a fixed prior.
Revised: Another difference lies in that optimizing a learnable functional prior is of most importance in NPs, while RWS algorithms are mostly used in scenarios with a fixed prior.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Technical Discussions.
Revised: Technical Discussions.

Original: In this paper, we study NPs family from an optimization objective perspective and analyze the inference suboptimality of vanilla NPs.
Revised: In this paper, we study NPs family from an optimization objective perspective and analyze the inference suboptimality of vanilla NPs.

Original: Within the variational expectation maximization framework, our developed SI-NP improves the target likelihood step by step to obtain a more optimal functional prior.
Revised: Within the variational expectation maximization framework, our developed SI-NP improves the target likelihood step by step to obtain a more optimal functional prior.

Original: Besides, experimental results tell us the learned functional prior has close connections with the number of context points and complexity of function families.
Revised: Besides, experimental results tell us the learned functional prior has close connections with the number of context points and complexity of function families.

Original: Existing Limitations.
Revised: Existing Limitations.

Original: Compared to vanilla NPs, SI-NP requires more than one Monte Carlo sample from the functional prior in training, which consumes more computations.
Revised: Compared to vanilla NPs, SI-NP requires more than one Monte Carlo sample from the functional prior in training, which consumes more computations.

Original: Though intuitively, the uncertainty of the functional prior can be forward propagated to the output distribution, the influence of such uncertainty has not been mathematically studied.
Revised: Though intuitively, the uncertainty of the functional prior can be forward propagated to the output distribution, the influence of such uncertainty has not been mathematically studied.

Original: Future Extensions.
Revised: Future Extensions.

Original: The SI-NP objective is regardless of neural architectures so that any structural inductive bias can be incorporated in modeling.
Revised: The SI-NP objective is regardless of neural architectures so that any structural inductive bias can be incorporated in modeling.

Original: The combination can theoretically produce superior functional representation baselines in the domain (Please refer to Appendix (H.5) for extensive experiments).
Revised: The combination can theoretically produce superior functional representation baselines in the domain (Please refer to Appendix (H.5) for extensive experiments).

Revised: A CKNOWLEDGEMENT

Revised: We thank Dharmesh Tailor for the helpful discussions.

Revised: We thank Eric Nalisnick and Tim Bakker for the helpful feedback in the AMLab Seminar talks.

Original: C ONTENTS 1 Introduction
Revised: C ONTENTS 1 Introduction

Original: 2 Preliminaries 3 Optimization Gaps and Statistical Traits 3.
Revised: 2 Preliminaries 3 Optimization Gaps and Statistical Traits 3.

Original: Inference Suboptimality in vanilla NPs . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Inference Suboptimality in vanilla NPs . . . . . . . . . . . . . . . . . . . . . . . .

Original: Evaluation Criteria & Asymptotic Performance . . . . . . . . . . . . . . . . . . .
Revised: Evaluation Criteria & Asymptotic Performance . . . . . . . . . . . . . . . . . . .

Original: 4 Tractable Optimization via Expectation Maximization 4.
Revised: 4 Tractable Optimization via Expectation Maximization 4.

Original: Variational Expectation Maximization for NPs . . . . . . . . . . . . . . . . . . . .
Revised: Variational Expectation Maximization for NPs . . . . . . . . . . . . . . . . . . . .

Original: Surrogate Function for Exact NPs . . . . . . . . . . . . . . . . . . . . . .
Revised: Surrogate Function for Exact NPs . . . . . . . . . . . . . . . . . . . . . .

Original: Tractable Optimization with Self-Normalized Importance Sampling . . . .
Revised: Tractable Optimization with Self-Normalized Importance Sampling . . . .

Original: Scalable Training and Testing . .
Revised: Scalable Training and Testing . .

Original: 5 Experiments and Analysis 5.
Revised: 5 Experiments and Analysis 5.

Original: Synthetic Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Synthetic Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: Image Completion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Image Completion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original:  Work 7 Conclusion
Revised: 6 Related Work 7 Conclusion

Original:  Frequently Asked Questions
Revised: A Frequently Asked Questions

Original: B Probabilistic Generative Process in NPs
Revised: B Probabilistic Generative Process in NPs

Original: C Predictive Distributions in GPs & NPs
Revised: C Predictive Distributions in GPs & NPs

Original: D NPs Formulation & Structural Inductive Biases
Revised: D NPs Formulation & Structural Inductive Biases

Original: D.1 Prior, Posterior & Proposal Distributions . .
Revised: D.1 Prior, Posterior & Proposal Distributions . .

Original: D.2 Approximate ELBOs in NPs & Proof of Remark 1 . . . . . . . . . . . . . . . . .
Revised: D.2 Approximate ELBOs in NPs & Proof of Remark 1 . . . . . . . . . . . . . . . . .

Original: D.3 Summary of NPs Family . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: D.3 Summary of NPs Family . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: D.4 Inference Gaps. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: D Inference Gaps

Original: E Formulation of Variational Expectation Maximization Method
Revised: E Formulation of Variational Expectation Maximization Method

Original: E.1 Proof of Improvement Guarantee using Variational Expectation Maximization . . .
Revised: E Proof of Improvement Guarantee using Variational Expectation Maximization . . .

Original: E.1.
Revised: E.1.

Original: Meta Learning Surrogate Function . . . . . . . . . . . . . . . . . . . . . .
Revised: Meta Learning Surrogate Function . . . . . . . . . . . . . . . . . . . . . .

Original: E.1.
Revised: E.1.

Original: Improvement Guarantee . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: Improvement Guarantee . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: E.2 ImportanceSampling in a Variational EM Algorithm . . . . . . . . . . . . . . . . 
Revised: E.2 Importance in a Variational EM Algorithm . . . . . . . . . . . . . . . . E.

Original: E.3 Optimization Objective with Proposal Distributions . . . . . . . . . . . . . . . . .
Revised: Objective with Proposal Distributions . . . . . . . . . . . . . . . . .

Original: E.3.
Revised: E.3.

Original: Update of Proposal Distributions (Optional) . . . . . . . . . . . . . . . . .
Revised: Update of Proposal Distributions (Optional) . . . . . . . . . . . . . . . . .

Original: E.3.Derivation
Revised: E.3.

Revised: [Table]

Original:  of the Proposal Update Objective . . . . . . . . . . . . . . . .
Revised: Derivation of the Proposal Update Objective . . . . . . . . . . . . . . . .

Original: E.4 Gradient Estimates in Variational EM . . . . . . . . . . . . . . . . . . . . . . . .
Revised: E.4 Gradient Estimates in Variational EM . . . . . . . . . . . . . . . . . . . . . . . .

Original: F Connection with CNPs
Revised: F Connection with CNPs

Original: F.
Revised: F.

Original: Prior Collapse in SI-NPs with One Monte Carlo Sample . . . . . . . . . . . . . . .
Revised: Prior Collapse in SI-NPs with One Monte Carlo Sample . . . . . . . . . . . . . . .

Original: F.
Revised: F.

Original: Effect of Adjusting Coefficients of the Prior Regularizer . . . . . . . . . . . . . .
Revised: Effect of Adjusting Coefficients of the Prior Regularizer . . . . . . . . . . . . . .

Original: G Experimental Setup & Implementation Details
Revised: G Experimental Setup & Implementation Details

Original: G.1 Meta Learning Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: G.1 Meta Learning Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: G.2 Neural Architectures & Optimizations & Evaluation Set-up . . . . . . . . . . . . .
Revised: G.2 Neural Architectures & Optimizations & Evaluation Set-up . . . . . . . . . . . . .

Original: H More Experimental Results
Revised: H More Experimental Results

Original: H.1 SI-NPs with A Learnable Proposal Distribution . . . . . . . . . . . . . . . . . . .
Revised: H.1 SI-NPs with A Learnable Proposal Distribution . . . . . . . . . . . . . . . . . . .

Original: H.2 Evaluation with More Monte Carlo Particles . . . . . . . . . . . . . . . . . . . . .
Revised: H.2 Evaluation with More Monte Carlo Particles . . . . . . . . . . . . . . . . . . . . .

Original: H.3 Influence of Dimensions of Latent Variables . . . . . . . . . . . . . . . . . . . . .
Revised: H.3 Influence of Dimensions of Latent Variables . . . . . . . . . . . . . . . . . . . . .

Original: H.4 Sim2Real Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: H.4 Sim2Real Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: H.5 Augmenting SI-NPs with Attention Networks . .
Revised: H.5 Augmenting SI-NPs with Attention Networks . .

Original: H.5.1 Synthetic Regression . .
Revised: H.5.1 Synthetic Regression . .

Original: H.5.2 Image Completion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Revised: H.5.2 Image Completion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Original: H.5.3 Sim2Real . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 
Revised: H.5.3 Sim2Real . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . H

Revised: [Table]

Original: H.6 Additional Visualizations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 
Revised: Additional Visualizations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . H

Original: H.6.1 More Synthetic Regression Related Results . . . . . . . . . . . . . . . . . 
Revised: [Table] Results . . . . . . . . . . . . . . . . . H

Original: H.6.2 More Image Completion Related Results . . . . . . . . . . . . . . . . . .
Revised: [Table] Related Results . . . . . . . . . . . . . . . . . .

Original: I Computation Tools
Revised: I Computation Tools

Original: A F REQUENTLY A SKED Q UESTIONS
Revised: A F REQUENTLY A SKED Q UESTIONS

Original: In this section, we list frequently asked questions from researchers who help proofread this manuscript.
Revised: In this section, we list frequently asked questions from researchers who help proofread this manuscript.

Original: These raised questions are crucial, so we include more detailed discussions here.
Revised: These raised questions are crucial, so we include more detailed discussions here.

Original: Meanwhile, we make extra efforts for readers to quickly understand our work and make slides as follows https://anonymous.4open.science/r/SI_NPs_Slides-7C94/iclr_ submission_slides.pdf .
Revised: Meanwhile, we make extra efforts for readers to quickly understand our work and make slides as follows https://anonymous.4open.science/r/SI_NPs_Slides-7C94/iclr_ submission_slides.pdf .

Original: Learnable Proposal Distributions.
Revised: Learnable Proposal Distributions.

Original: The reason why we set the update of learnable proposal distribution an optional choice is that we find it difficult to balance the optimization of Eq.
Revised: The reason why we set the update of learnable proposal distribution an optional choice is that we find it difficult to balance the optimization of Eq.

Original: (12) and Eq.
Revised: (12) and Eq.

Original: This is non-trivial and assigning equal weights in optimization results in unstable performance in experiments (Please refer to Section (H.1) for more discussion.).
Revised: This is non-trivial and assigning equal weights in optimization results in unstable performance in experiments (Please refer to Section (H.1) for more discussion.).

Original: So, we leave the improvement of the proposal distribution future research.
Revised: So, we leave the improvement of the proposal distribution future research.

Original: Suppose a reasonable objective or at least a weight scheduling mechanism for two objectives in two E -steps can be developed to stabilize the training.
Revised: Suppose a reasonable objective or at least a weight scheduling mechanism for two objectives in two E -steps can be developed to stabilize the training.

Original: In that case, SI-NPs are likely to achieve even better performance.
Revised: In that case, SI-NPs are likely to achieve even better performance.

Original: Influence of Functional Prior Collapse.
Revised: Influence of Functional Prior Collapse.

Original: Generally, the generated functions are more diverse when the functional prior does not collapse: More randomness can be reflected from the functional prior, and the uncertainty of partial observation can be propagated to the output space.
Revised: Generally, the generated functions are more diverse when the functional prior does not collapse: More randomness can be reflected from the functional prior, and the uncertainty of partial observation can be propagated to the output space.

Original: However, in experiments, we observe that for functions with more superficial structures, e.g. MNIST dataset, the deterministic functional representation in CNPs is sufficient to generate functions of high quality.
Revised: However, in experiments, we observe that for functions with more superficial structures, e.g. MNIST dataset, the deterministic functional representation in CNPs is sufficient to generate functions of high quality.

Original: So there is no consensus on the influence of the functional prior collapse on the generation performance.
Revised: So there is no consensus on the influence of the functional prior collapse on the generation performance.

Original: Influence of the Number of Inference Particles in SI-NPs.
Revised: Influence of the Number of Inference Particles in SI-NPs.

Original: With more inference particles (latent variables), SI-NPs can avoid the prior collapse in some cases, but the cost is expensive for computations.
Revised: With more inference particles (latent variables), SI-NPs can avoid the prior collapse in some cases, but the cost is expensive for computations.

Original: To trade-off performance and computations, we set the required number of particles for meta training in a small quantity, e.g. 8 particles for image completion tasks.
Revised: To trade-off performance and computations, we set the required number of particles for meta training in a small quantity, e.g. 8 particles for image completion tasks.

Original: Connections between Different Optimization Objectives.
Revised: Connections between Different Optimization Objectives.

Original: In Table (3), we summarize the connections of different objectives with the meta dataset log-likelihood L ( ϑ ) .
Revised: In Table (3), we summarize the connections of different objectives with the meta dataset log-likelihood L ( ϑ ) .

Original: Note that L NP ( ϑ, ϕ ) is not the exact ELBO of L ( ϑ ) .
Revised: Note that L NP ( ϑ, ϕ ) is not the exact ELBO of L ( ϑ ) .

Original: L ML-NP ( ϑ ) can also be viewed as the importance weighted method with equal weights for all particles.
Revised: L ML-NP ( ϑ ) can also be viewed as the importance weighted method with equal weights for all particles.

Original: Since the estimates of importance weights in SI-NPs exploit the target observations and consider the difference in particles, this helps improve the model’s generalization capability.
Revised: Since the estimates of importance weights in SI-NPs exploit the target observations and consider the difference in particles, this helps improve the model’s generalization capability.

Original: Though the self-normalized importance sampling makes SI-NPs’ estimates over
Revised: Though the self-normalized importance sampling makes SI-NPs’ estimates over

Original: L ( ϑ ) biased, there is a probabilistic convergence guarantee towards the true integral quantity of our interest (Please refer to Chapter 9 in the Book ” Monte Carlo Theory, Methods and Examples ” (Owen, 2013)).
Revised: L ( ϑ ) biased, there is a probabilistic convergence guarantee towards the true integral quantity of our interest (Please refer to Chapter 9 in the Book ” Monte Carlo Theory, Methods and Examples ” (Owen, 2013)).

Original: Combination with Structural Inductive Biases.
Revised: Combination with Structural Inductive Biases.

Original: The primary research focus is on the optimality of the different optimization objectives, and this study applies to more general stochastic processes (stationary or nonstationary cases).
Revised: The primary research focus is on the optimality of the different optimization objectives, and this study applies to more general stochastic processes (stationary or nonstationary cases).

Original: Theoretically, the objective of SI-NPs can be combined with most structural inductive biases in Table (4), especially when the function family is complicated.
Revised: Theoretically, the objective of SI-NPs can be combined with most structural inductive biases in Table (4), especially when the function family is complicated.

Original: For example, the scale parameter of Matern kernels also varies with kernel values, bringing more variations in realizations.
Revised: For example, the scale parameter of Matern kernels also varies with kernel values, bringing more variations in realizations.

Original: In this case, we guess a global latent variable is difficult to handle: SI-NPs and ML-NPs might encounter a similar performance bottleneck from the empirical observations in the main paper.
Revised: In this case, we guess a global latent variable is difficult to handle: SI-NPs and ML-NPs might encounter a similar performance bottleneck from the empirical observations in the main paper.

Original: To further improve the performance, we take the attention module (Kim et al., 2019) as an example to conduct extensive experiments in Section (H.5).
Revised: To further improve the performance, we take the attention module (Kim et al., 2019) as an example to conduct extensive experiments in Section (H.5).

Original: Since the meta learning experiment is computationally expensive and time-consuming in training processes, we do not examine combinations with other inductive biases in this paper.
Revised: Since the meta learning experiment is computationally expensive and time-consuming in training processes, we do not examine combinations with other inductive biases in this paper.

Original: GP Oracles in Synthetic Regression.
Revised: GP Oracles in Synthetic Regression.

Original: Note that the GP oracle is computed in the form ln p ( y 1: n + m | y 1: n , x 1: n + m ) , where the predictive distribution is mainly with a non-diagonal covariance matrix (suggesting y n + m not independent in statistics).
Revised: Note that the GP oracle is computed in the form ln p ( y 1: n + m | y 1: n , x 1: n + m ) , where the predictive distribution is mainly with a non-diagonal covariance matrix (suggesting y n + m not independent in statistics).

Original: In comparison, the log-likelihood of
Revised: In comparison, the log-likelihood of

Original: NPs family is computed in the form (cid:80) n + m i
Revised: NPs family is computed in the form (cid:80) n + m i

Original: =1 ln p ( y i |
Revised: =1 ln p ( y i |

Original: [ x i , z ]; ϑ ) , where y 1: n + m is conditional independent w.r.t.
Revised: [ x i , z ]; ϑ ) , where y 1: n + m is conditional independent w.r.t.

Original: the global latent variable z .
Revised: the global latent variable z .

Original: Such a difference in computations causes the mentioned gap.
Revised: Such a difference in computations causes the mentioned gap.

Original: However, with a large neural model, such as the integration of attention networks in Section (H.5), the quantified performance gap of GP Oracles and the log-likelihood of NPs family is well reduced.[Table] [Equation] [Table]
Revised: However, with a large neural model, such as the integration of attention networks in Section (H.5), the quantified performance gap of GP Oracles and the log-likelihood of NPs family is well reduced.

Original: [Equation]

Original: [Table]
Revised: [Table]

Original: B P ROBABILISTIC G ENERATIVE P ROCESS IN NP S
Revised: B P ROBABILISTIC G ENERATIVE P ROCESS IN NP S

Original: Definition B.1 (Exchangeable Stochastic Processes)
Revised: Definition B.1 (Exchangeable Stochastic Processes)

Original: We denote a probability space of functions by (Ω , F , P ) .
Revised: We denote a probability space of functions by (Ω , F , P ) .

Original: Let ν x 1 ,...,x N be a probability measure on R d with { x 1 , . . .
Revised: Let ν x 1 ,...,x N be a probability measure on R d with { x 1 , . . .

Original: , x N } a finite index set.
Revised: , x N } a finite index set.

Original: The process is called an exchangeable stochastic process S : X × Ω → R d such that ν x 1 ,...,x N ( F 1 ×· · ·×
Revised: The process is called an exchangeable stochastic process S : X × Ω → R d such that ν x 1 ,...,x N ( F 1 ×· · ·×

Original: F N ) =
Revised: F N ) =

Original: P ( S x 1 ∈ F 1 , . .
Revised: P ( S x 1 ∈ F 1 , . .

Original: S x N ∈ F N ) if it satisfies exchangeable consistency and marginalization consistency.
Revised: S x N ∈ F N ) if it satisfies exchangeable consistency and marginalization consistency.

Original: Here we can translate the generative process of NPs in the following mathematical way.
Revised: Here we can translate the generative process of NPs in the following mathematical way.

Original: [Equation]
Revised: [Equation]

Original: C P REDICTIVE D ISTRIBUTIONS IN GP S & NP S
Revised: C P REDICTIVE D ISTRIBUTIONS IN GP S & NP S

Original: Here we take one-dimensional deep Gaussian processes (Dai et al., 2016) as an example.
Revised: Here we take one-dimensional deep Gaussian processes (Dai et al., 2016) as an example.

Original: With context points D C
Revised: With context points D C

Original: = { ( x i , y i ) } ni =1 and target points D T
Revised: = { ( x i , y i ) } ni =1 and target points D T

Original: = { ( x i , y i ) } n + m i =
Revised: = { ( x i , y i ) } n + m i =

Original: =
Revised: =

Original: [ x T , y T ] , the key to applications is the predictive distribution p ( f ( x T )
Revised: [ x T , y T ] , the key to applications is the predictive distribution p ( f ( x T )

Original: |D C , x T ) =
Revised: |D C , x T ) =

Original: N ( y T ; µ T , Σ T ) .
Revised: N ( y T ; µ T , Σ T ) .

Original: The conditional mean µ T and covariance Σ T functions in Eq.
Revised: The conditional mean µ T and covariance Σ T functions in Eq.

Original: (16) are permutation invariant to the order of context points.
Revised: (16) are permutation invariant to the order of context points.

Original: [Equation]
Revised: [Equation]

Original: Here the covariance matrix denoted by Σ is computed with the context input x C =
Revised: Here the covariance matrix denoted by Σ is computed with the context input x C =

Original: ( x 1 , . .
Revised: ( x 1 , . .

Original: , x n ) ∈
Revised: , x n ) ∈

Original: R n × d , the target input x T = ( x 1 , . . .
Revised: R n × d , the target input x T = ( x 1 , . . .

Original: , x n + m ) ∈ R ( n + m ) × d , and a kernel function ψ , e.g.
Revised: , x n + m ) ∈ R ( n + m ) × d , and a kernel function ψ , e.g.

Original: [Σ C,C ] i,j = ψ ( x i , x j ) , m θ is the mean function m θ , and the context output is y C
Revised: [Σ C,C ] i,j = ψ ( x i , x j ) , m θ is the mean function m θ , and the context output is y C

Original: = ( y 1 , . .
Revised: = ( y 1 , . .

Original: y n ) ∈ R n .
Revised: y n ) ∈ R n .

Original: Nevertheless, the computation of matrix inversion in Eq.
Revised: Nevertheless, the computation of matrix inversion in Eq.

Original: (16) makes the runtime complexity as expensive as O (( n + m ) 3 ) .
Revised: (16) makes the runtime complexity as expensive as O (( n + m ) 3 ) .

Original: D NP S F ORMULATION & S TRUCTURAL I NDUCTIVE B IASES D. P RIOR , P OSTERIOR & P ROPOSAL D ISTRIBUTIONS
Revised: D NP S F ORMULATION & S TRUCTURAL I NDUCTIVE B IASES D. P RIOR , P OSTERIOR & P ROPOSAL D ISTRIBUTIONS

Original: Since fast adaptation is achieved in an amortized way, which reduces the gradient updates w.r.t.
Revised: Since fast adaptation is achieved in an amortized way, which reduces the gradient updates w.r.t.

Original: model parameters to learning function specific latent variables with meta trained neural networks.
Revised: model parameters to learning function specific latent variables with meta trained neural networks.

Original: Definition D.1 (Permutation Invariant Functions) Let S n be an n -element permutation group.
Revised: Definition D.1 (Permutation Invariant Functions) Let S n be an n -element permutation group.

Original: For any permutation operator g ∈ S n , the function is a bijective mapping from the order set { 1 , 2 , . . . , n } to itself:
Revised: For any permutation operator g ∈ S n , the function is a bijective mapping from the order set { 1 , 2 , . . . , n } to itself:

Original: [Equation]
Revised: [Equation]

Original: Then given a set of data points { x 1 , x 2 , . . .
Revised: Then given a set of data points { x 1 , x 2 , . . .

Original: , x n } , the function Φ is said to be a permutation invariant function if the following equation is satisfied ∀ g ∈ S n 
Revised: , x n } , the function Φ is said to be a permutation invariant function if the following equation is satisfied ∀ g ∈ S n [Equation] , x

Original: [Equation]
Revised: [Equation]

Revised: , x n

Revised: [Equation]

Original: ]) =
Revised: ) =

Original: Φ([ x g 1 , x g 2 , . .
Revised: Φ([ x g 1 , x g 2 , . .

Original: , x g n ]) =
Revised: , x g n ]) =

Original: Φ([ x 1 , x 2 , . . .
Revised: Φ([ x 1 , x 2 , . . .

Original: , x n ]) .
Revised: , x n ]) .

Original: The context points and the target points are treated as sets, so the amortized network should be permutation invariant w.r.t.
Revised: The context points and the target points are treated as sets, so the amortized network should be permutation invariant w.r.t.

Original: the order of data points.
Revised: the order of data points.

Original: Approximate Posterior Distribution.
Revised: Approximate Posterior Distribution.

Original: This is denoted by q ϕ ( z |D Tτ ) in this paper.
Revised: This is denoted by q ϕ ( z |D Tτ ) in this paper.

Original: Usually, the approximate posterior is used in NPs (Garnelo et al., 2018b; Kim et al., 2019) and works as a proxy for the non-analytical exact posterior p ( z |D Tτ ; ϑ ) .
Revised: Usually, the approximate posterior is used in NPs (Garnelo et al., 2018b; Kim et al., 2019) and works as a proxy for the non-analytical exact posterior p ( z |D Tτ ; ϑ ) .

Original: Prior Distribution.
Revised: Prior Distribution.

Original: This is denoted by p ( z |D
Revised: This is denoted by p ( z |D

Original: Cτ ; ϑ ) in this paper.
Revised: Cτ ; ϑ ) in this paper.

Original: Unlike the approximate prior q ϕ ( z |D Cτ ) used in NPs, we use an exact functional prior in SI-NPs.
Revised: Unlike the approximate prior q ϕ ( z |D Cτ ) used in NPs, we use an exact functional prior in SI-NPs.

Original: Proposal Distribution.
Revised: Proposal Distribution.

Original: This is denoted by q η ( z |D Tτ ) in this paper.
Revised: This is denoted by q η ( z |D Tτ ) in this paper.

Original: The role of the proposal distribution resembles that of the approximate posterior in NPs.
Revised: The role of the proposal distribution resembles that of the approximate posterior in NPs.

Original: It is used to sample latent variables and enables the computation of the importance weights in NPs.
Revised: It is used to sample latent variables and enables the computation of the importance weights in NPs.

Original: [Figure]
Revised: [Figure]

Original: In vanilla NPs, the latent variable z is inferred from a set of data points, and the distributions q ϕ
Revised: In vanilla NPs, the latent variable z is inferred from a set of data points, and the distributions q ϕ

Original: ( z ) and q ϕ ( z |D Cτ ) are learned via functions permutation invariant to the order of data points.
Revised: ( z ) and q ϕ ( z |D Cτ ) are learned via functions permutation invariant to the order of data points.

Original: An example module to parameterize q ϕ ( z |D Cτ ) =
Revised: An example module to parameterize q ϕ ( z |D Cτ ) =

Original: N ( z ; µ C , Σ C ) can be Eq.
Revised: N ( z ; µ C , Σ C ) can be Eq.

Original: (17) with (cid:76) a mean pooling operator and { h ϕ , g ϕ } encoder networks.
Revised: (17) with (cid:76) a mean pooling operator and { h ϕ , g ϕ } encoder networks.

Original: [Equation]
Revised: [Equation]

Original: The same with that in traditional stochastic processes, a realisation corresponds to a sampled function f
Revised: The same with that in traditional stochastic processes, a realisation corresponds to a sampled function f

Original: ( X ) generated in a sequential way:
Revised: ( X ) generated in a sequential way:

Original: z ∼ q ϕ ( z |D Cτ ) , f ( X ) ∼ p
Revised: z ∼ q ϕ ( z |D Cτ ) , f ( X ) ∼ p

Original: ( Y | X, z ; ϑ ) .
Revised: ( Y | X, z ; ϑ ) .

Original: Since this paper aims to study the influence of different optimization objectives concerning the NP models, we retain the basic model set-up for all baselines.
Revised: Since this paper aims to study the influence of different optimization objectives concerning the NP models, we retain the basic model set-up for all baselines.

Original: The Fig.
Revised: The Fig.

Original: (7) exhibits the neural architecture of the vanilla NPs, and such a structure is also shared with ML-NPs and SI-NPs in our paper.
Revised: (7) exhibits the neural architecture of the vanilla NPs, and such a structure is also shared with ML-NPs and SI-NPs in our paper.

Original: As previously mentioned, these methods can be straightforwardly incorporated with other structural inductive biases (Kim et al., 2019; Gordon et al., 2019; Gondal et al., 2021; Wang & van Hoof, 2022).
Revised: As previously mentioned, these methods can be straightforwardly incorporated with other structural inductive biases (Kim et al., 2019; Gordon et al., 2019; Gondal et al., 2021; Wang & van Hoof, 2022).

Original: For example, in the following section, we augment all methods with attention networks (Kim et al., 2019) and examine the resulting performance.
Revised: For example, in the following section, we augment all methods with attention networks (Kim et al., 2019) and examine the resulting performance.

Original: D. A PPROXIMATE ELBO S IN NP S & P ROOF OF R EMARK
Revised: D. A PPROXIMATE ELBO S IN NP S & P ROOF OF R EMARK

Original: [Equation]
Revised: [Equation]

Original: For Eq.
Revised: For Eq.

Original: (18.b), remember that w.r.t.
Revised: (18.b), remember that w.r.t.

Original: these VAE-like methods, it is hard to guarantee the improvement of the evidence in each iteration when optimizing ELBO due to the existence of a posterior approximation gap.
Revised: these VAE-like methods, it is hard to guarantee the improvement of the evidence in each iteration when optimizing ELBO due to the existence of a posterior approximation gap.

Original: Meanwhile, the form of the functional prior is unknown.
Revised: Meanwhile, the form of the functional prior is unknown.

Original: Proof D.1 (Remark 1) Vanilla NPs directly replace the real functional prior by the approximate one q ϕ ( z |D Cτ ) and introduce the consistent regularizer in Eq.
Revised: Proof D.1 (Remark 1) Vanilla NPs directly replace the real functional prior by the approximate one q ϕ ( z |D Cτ ) and introduce the consistent regularizer in Eq.

Original: We further introduce the prior approximation gap in Eq.
Revised: We further introduce the prior approximation gap in Eq.

Original: (19), in which the sign is undetermined.
Revised: (19), in which the sign is undetermined.

Original: [Equation]
Revised: [Equation]

Original: Based on decomposition in Eq.
Revised: Based on decomposition in Eq.

Original: (6)/(19), we can find that there exists no consistent monotonic relationship between L ( ϑ ) and L NP ( ϑ, ϕ ) .
Revised: (6)/(19), we can find that there exists no consistent monotonic relationship between L ( ϑ ) and L NP ( ϑ, ϕ ) .

Original: The invalid ELBO makes the optimization w.r.t.
Revised: The invalid ELBO makes the optimization w.r.t.

Original: L NP ( ϑ, ϕ ) not always improve log-likelihood L ( ϑ ) .
Revised: L NP ( ϑ, ϕ ) not always improve log-likelihood L ( ϑ ) .

Original: [Equation]
Revised: [Equation]

Original: On the left side of Eq.
Revised: On the left side of Eq.

Original: (20), the ELBO of NPs can be bounded by the log-likelihood of meta learning dataset.
Revised: (20), the ELBO of NPs can be bounded by the log-likelihood of meta learning dataset.

Original: Optimizing the ELBO can guarantee the optimal or local optimal solution when the approximation gap and the amortization gap are well closed.
Revised: Optimizing the ELBO can guarantee the optimal or local optimal solution when the approximation gap and the amortization gap are well closed.

Original: However, the right side of Eq.
Revised: However, the right side of Eq.

Original: (20) implies the previous commonly-used strategies, e.g. normalizing flows for richer variational posterior distributions (Rezende & Mohamed, 2015), auxiliary variables for augmented variational posterior distributions (Maaløe et al., 2016) and more flexible prior distributions (Tomczak & Welling, 2018), to close VAEs inference gaps (Cremer et al., 2018) will not guarantee the performance improvement in a theoretical sense.
Revised: (20) implies the previous commonly-used strategies, e.g. normalizing flows for richer variational posterior distributions (Rezende & Mohamed, 2015), auxiliary variables for augmented variational posterior distributions (Maaløe et al., 2016) and more flexible prior distributions (Tomczak & Welling, 2018), to close VAEs inference gaps (Cremer et al., 2018) will not guarantee the performance improvement in a theoretical sense.

Original: In other words, the consistent regularizer in NPs is ill-posed for optimization.
Revised: In other words, the consistent regularizer in NPs is ill-posed for optimization.

Original: D. S UMMARY OF NP S F AMILY
Revised: D. S UMMARY OF NP S F AMILY

Original: In this part, we summarize the encoder and decoders for typical NP variants and point out the inductive biases behind them.
Revised: In this part, we summarize the encoder and decoders for typical NP variants and point out the inductive biases behind them.

Original: The information is summarized in Table (4).
Revised: The information is summarized in Table (4).

Original: We can see a variety of inductive biases can be used to improve NPs, most of them from the model structure perspective rather than the optimization objective.
Revised: We can see a variety of inductive biases can be used to improve NPs, most of them from the model structure perspective rather than the optimization objective.

Original: [Table]
Revised: [Table]

Original: D. I NFERENCE G APS
Revised: D. I NFERENCE G APS

Original: In this section
Revised: In this section

Original: [Table]
Revised: [Table]

Original: we apply the trick of inference gap decomposition (Cremer et al., 2018) to understand vanilla NPs.
Revised: we apply the trick of inference gap decomposition (Cremer et al., 2018) to understand vanilla NPs.

Original: Here we denote the approximate inference gap by D AI KL and the posterior approximation gap by D PA KL in Table (5).
Revised: Here we denote the approximate inference gap by D AI KL and the posterior approximation gap by D PA KL in Table (5).

Original: Since it is intractable to obtain the exact form for the functionalprior and the posterior, we cannot directly close the mentioned approximate gap, and approximate methods are used to learn them.
Revised: Since it is infeasible to obtain the exact form for the functional posterior, we cannot directly close the mentioned approximate gap, and approximate methods are used to learn them.

Original: For example, in vanilla NPs, the consistent regularizer in Eq.
Revised: In vanilla NPs, the consistent regularizer in Eq.

Original: (7) works as the surrogate for the approximate gap.
Revised: (7) works as the surrogate for the ELBO.

Original: But based on our analysis, closing this surrogate gap cannot lead to a theoretically optimal solution of functional priors.
Revised: But based on our analysis, closing this surrogate gap cannot lead to a theoretically optimal solution of functional priors.

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table][Equation] [Table] [Equation]
Revised: [Table]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table][Equation] [Table] [Equation]
Revised: [Table]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: E F ORMULATION OF V ARIATIONAL E XPECTATION M AXIMIZATION M ETHOD
Revised: E F ORMULATION OF V ARIATIONAL E XPECTATION M AXIMIZATION M ETHOD

Original: In this section, we detail the progress of optimizing the NP model with the help of variational expectation maximization algorithms.
Revised: In this section, we detail the progress of optimizing the NP model with the help of variational expectation maximization algorithms.

Original: The concept of meta learning surrogate functions is introduced, and NPs are verified.
Revised: The concept of meta learning surrogate functions is introduced, and NPs are verified.

Original: Meanwhile, the improvement guarantee as well as other concerning technical points are included to better understand our method.
Revised: Meanwhile, the improvement guarantee as well as other concerning technical points are included to better understand our method.

Original: E. P ROOF OF I MPROVEMENT G UARANTEE USING V ARIATIONAL E XPECTATION M AXIMIZATION E.1. M ETA L EARNING S URROGATE F UNCTION
Revised: E. P ROOF OF I MPROVEMENT G UARANTEE USING V ARIATIONAL E XPECTATION M AXIMIZATION E.1. M ETA L EARNING S URROGATE F UNCTION

Original: Definition E.1 (Surrogate Function) 

Revised: Definition E.1 (Surrogate Function)

Original: Given the objective function f ( ϑ ) to maximize, surrogate functions g ( ϑ ; ϑ k ) w.r.t.
Revised: Given the objective function f ( ϑ ) to maximize, surrogate functions g ( ϑ ; ϑ k ) w.r.t.

Original: f ( ϑ ) are a family of functions with the following properties
Revised: f ( ϑ ) are a family of functions with the following properties

Original: • Both f ( ϑ ) and g ( ϑ ; ϑ k ) are C 2 -functions, which means f ( ϑ ) has first order and second order derivatives of those functions exist and these are smooth in the domain of the function.
Revised: • Both f ( ϑ ) and g ( ϑ ; ϑ k ) are C 2 -functions, which means f ( ϑ ) has first order and second order derivatives of those functions exist and these are smooth in the domain of the function.

Original: • The following two formulas hold:
Revised: • The following two formulas hold:

Original: [Equation]
Revised: [Equation]

Original: When the objective function f ( ϑ ) to maximize is complicated, e.g. multi-modal likelihood functions, a surrogate function g ( ϑ ; ϑ k ) enables an easier proxy implementation with convergence guarantee to at least the local optimal.
Revised: When the objective function f ( ϑ ) to maximize is complicated, e.g. multi-modal likelihood functions, a surrogate function g ( ϑ ; ϑ k ) enables an easier proxy implementation with convergence guarantee to at least the local optimal.

Original: To see this point, recall the properties of the surrogate function g ( ϑ k ; ϑ k )
Revised: To see this point, recall the properties of the surrogate function g ( ϑ k ; ϑ k )

Original: =
Revised: =

Original: f
Revised: f

Original: ( ϑ k ) .
Revised: ( ϑ k ) .

Original: The update rule for the surrogate function follows that ϑ k +1 = arg max ϑ g ( ϑ ; ϑ k ) .
Revised: The update rule for the surrogate function follows that ϑ k +1 = arg max ϑ g ( ϑ ; ϑ k ) .

Original: And this results in f ( ϑ k +1 ) ≥ g ( ϑ k +1 ; ϑ k ) ≥ f
Revised: And this results in f ( ϑ k +1 ) ≥ g ( ϑ k +1 ; ϑ k ) ≥ f

Original: ( ϑ k ) .
Revised: ( ϑ k ) .

Original: Recall the following function L ( ϑ ; ϑ k ) in the main paper.
Revised: Recall the following function L ( ϑ ; ϑ k ) in the main paper.

Original: [Equation]
Revised: [Equation]

Original: The expectation operation in the k -th iteration step corresponds to E-step : q ϕ ( z |D Cτ ) = p ϑ k ( z |D Cτ ) , while the maximization operation in the k -th iteration step updates the parameter as
Revised: The expectation operation in the k -th iteration step corresponds to E-step : q ϕ ( z |D Cτ ) = p ϑ k ( z |D Cτ ) , while the maximization operation in the k -th iteration step updates the parameter as

Original: M-step : ϑ 
Revised: M-step : ϑ k

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: ( ϑ ; ϑ k ) .
Revised: ϑ ; ϑ k ) .

Revised: [Equation]

Revised: Model Evidence

Revised: [Equation]

Revised: Model Evidence □

Original: [Equation]
Revised: [Equation]

Original: E.1. I MPROVEMENT G UARANTEE
Revised: E.1. I MPROVEMENT G UARANTEE

Original: As illustrated in Eq.s (23), the surrogate function L ( ϑ ; ϑ k ) is bounded by two log-likelihoods.
Revised: As illustrated in Eq.s (23), the surrogate function L ( ϑ ; ϑ k ) is bounded by two log-likelihoods.

Original: Over the process of iterations, the log-likelihood is gradually increased to the final convergence.
Revised: Over the process of iterations, the log-likelihood is gradually increased to the final convergence.

Original: [Equation]
Revised: [Equation]

Original: This indicates that the finally updated parameters of the surrogate function are exactly (sub-)optimal ones for the evidence.
Revised: This indicates that the finally updated parameters of the surrogate function are exactly (sub-)optimal ones for the evidence.

Original: Based on these rules, directly optimizing the surrogate function step by step can theoretically guarantee the finding of optimal or at least a local optimal parameters.
Revised: Based on these rules, directly optimizing the surrogate function step by step can theoretically guarantee the finding of optimal or at least a local optimal parameters.

Original: E. I MPORTANCE S AMPLING IN A V ARIATIONAL EM A LGORITHM
Revised: E. I MPORTANCE S AMPLING IN A V ARIATIONAL EM A LGORITHM

Original: Though the conditional marginal distribution p ( D Tτ |D
Revised: Though the conditional marginal distribution p ( D Tτ |D

Original: Cτ ; ϑ k ) is not analytical, the importance sampling trick can be used to estimate the result with the help of a proposal distribution q η ( z |D Tτ ) .
Revised: Cτ ; ϑ k ) is not analytical, the importance sampling trick can be used to estimate the result with the help of a proposal distribution q η ( z |D Tτ ) .

Original: [Equation]
Revised: [Equation]

Original: with z ( b ) ∼ q η ( z |D Tτ ) and ω ( b ) = p( D
Revised: with z ( b ) ∼ q η ( z |D Tτ ) and ω ( b ) = p

Original: [Equation]
Revised: [Equation]

Original: , z (

Original: [Equation]

Original: [Equation] [Equation] [Equation] Especially, the joint distribution is computed via the decomposition that p ( D Tτ , z ( b ) |D
Revised: Especially, the joint distribution is computed via the decomposition that p ( D Tτ , z ( b ) |D

Original: Cτ ; ϑ k ) =
Revised: Cτ ; ϑ k ) =

Original: p ( z ( b ) |D
Revised: p ( z ( b ) |D

Original: Cτ ; ϑ k ) p ( D Tτ | z ( b ) ; ϑ k ) with p ( D Tτ | z ( b ) ; ϑ k ) =
Revised: Cτ ; ϑ k ) p ( D Tτ | z ( b ) ; ϑ k ) with p ( D Tτ | z ( b ) ; ϑ k ) =

Original: (cid:81) n + m i =1 p
Revised: (cid:81) n + m i =1 p

Original: y i |
Revised: y i |

Original: [ x i , z ( b ) ]; ϑ k ) .
Revised: [ x i , z ( b ) ]; ϑ k ) .

Original: With the above equation, the intractable optimization objective is transformed into a feasible one.
Revised: With the above equation, the intractable optimization objective is transformed into a feasible one.

Original: [Equation]
Revised: [Equation]

Original: We can expand the term inside the expectation in Eq (26.a) as follows.
Revised: We can expand the term inside the expectation in Eq (26.a) as follows.

Original: [Equation]
Revised: [Equation]

Original: As for the exact posterior p ( z |D Tτ ; ϑ k )
Revised: As for the exact posterior p ( z |D Tτ ; ϑ k )

Original: [Equation]
Revised: [Equation]

Original: we can get the following expansion.
Revised: we can get the following expansion.

Original: [Equation]
Revised: [Equation]

Original: The distribution has the complicated denominator and this makes it infeasible to directly sample from the conditional distribution.
Revised: The distribution has the complicated denominator and this makes it infeasible to directly sample from the conditional distribution.

Original: E. O PTIMIZATION O BJECTIVE WITH P ROPOSAL D ISTRIBUTIONS
Revised: E. O PTIMIZATION O BJECTIVE WITH P ROPOSAL D ISTRIBUTIONS

Original: This subsection shows the mentioned optional optimization step E -step #2 in Algorithm (1).
Revised: This subsection shows the mentioned optional optimization step E -step #2 in Algorithm (1).

Original: E.3. U PDATE OF P ROPOSAL D ISTRIBUTIONS (O PTIONAL )
Revised: E.3. U PDATE OF P ROPOSAL D ISTRIBUTIONS (O PTIONAL )

Original: The use of proposal distribution q η enables sampling z for Monte Carlo estimates.
Revised: The use of proposal distribution q η enables sampling z for Monte Carlo estimates.

Original: Another role of the proposal distribution is to work as a proxy for the posterior p ( z |D Tτ ; ϑ k ) , and the variance
Revised: Another role of the proposal distribution is to work as a proxy for the posterior p ( z |D Tτ ; ϑ k ) , and the variance

Original: V
Revised: V

Original: [Equation]
Revised: [Equation]

Original: p
Revised: Tτ ,

Original: [Equation]
Revised: [Equation]

Original: D Tτ , z |D
Revised: |D

Original: Cτ ; ϑ ) (cid:105) is expected to be lower for stable training.
Revised: Cτ ; ϑ ) (cid:105) is expected to be lower for stable training.

Original: So a reasonable optimization objective is to get two distributions as close as each other, e.g. minimizing the Kullback–Leibler divergence D KL [ p ( z |D Tτ ; ϑ k ) ∥
Revised: So a reasonable optimization objective is to get two distributions as close as each other, e.g. minimizing the Kullback–Leibler divergence D KL [ p ( z |D Tτ ; ϑ k ) ∥

Original: q η ( z |D Tτ )] .
Revised: q η ( z |D Tτ )] .

Original: This treatment is equivalent to wake phase updates in (Bornschein & Bengio, 2014).
Revised: This treatment is equivalent to wake phase updates in (Bornschein & Bengio, 2014).

Original: We can obtain the optimization objective on the right side of Eq.
Revised: We can obtain the optimization objective on the right side of Eq.

Original: (29) with the help of the self-normalized importance sampling.
Revised: (29) with the help of the self-normalized importance sampling.

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Here the self-normalized importance weights { ˆ ω ( b ) }
Revised: Here the self-normalized importance weights { ˆ ω ( b ) }

Original: Bb =1 inside the above equation are the same as that in Eq.
Revised: Bb =1 inside the above equation are the same as that in Eq.

Original: Also note that the the denominator 1 B (cid:80)
Revised: Also note that the the denominator 1 B (cid:80)

Original: Bb ′
Revised: Bb ′

Original: =1 ω ( b ′ ) of the weight relates to the estimate of p ( D Tτ |D
Revised: =1 ω ( b ′ ) of the weight relates to the estimate of p ( D Tτ |D

Original: Cτ ) , which has biases with limited samples at the beginning of training.
Revised: Cτ ) , which has biases with limited samples at the beginning of training.

Original: However, with the improvement of approximation, the bias can be decreased accordingly (Zimmermann et al., 2021).
Revised: However, with the improvement of approximation, the bias can be decreased accordingly (Zimmermann et al., 2021).

Original: E.3. D ERIVATION OF THE P ROPOSAL U PDATE O BJECTIVE
Revised: E.3. D ERIVATION OF THE P ROPOSAL U PDATE O BJECTIVE

Original: It is trivial to see the following equation since the term E p ( z |D Tτ ;
Revised: It is trivial to see the following equation since the term E p ( z |D Tτ ;

Original: ϑ k ) (cid:2) p ( z |D Tτ ; ϑ k ) (cid:3) is a constant.
Revised: ϑ k ) (cid:2) p ( z |D Tτ ; ϑ k ) (cid:3) is a constant.

Original: [Equation]
Revised: [Equation]

Original: Once again, we apply self-normalized importance sampling to the right side of Eq.
Revised: Once again, we apply self-normalized importance sampling to the right side of Eq.

Original: With the same set of sampled latent variables, the reweighted objective w.r.t.
Revised: With the same set of sampled latent variables, the reweighted objective w.r.t.

Original: the proposal distribution can be derived.
Revised: the proposal distribution can be derived.

Original: [Equation]
Revised: [Equation]

Original: E. G RADIENT E STIMATES IN V ARIATIONAL EM
Revised: E. G RADIENT E STIMATES IN V ARIATIONAL EM

Original: In the E-step #2, note that the model parameter is fixed as ϑ k , we can estimate the gradient of η w.r.t.
Revised: In the E-step #2, note that the model parameter is fixed as ϑ k , we can estimate the gradient of η w.r.t.

Original: L KL ( η ; η k − 1 , ϑ k ) in the following way.
Revised: L KL ( η ; η k − 1 , ϑ k ) in the following way.

Original: This operation is to close the divergence between the proposal distribution and the exact posterior distribution.
Revised: This operation is to close the divergence between the proposal distribution and the exact posterior distribution.

Original: [Equation]
Revised: [Equation]

Original: In the M-step , note that the normalized importance weights are constant, the proposal distribution is fixed [Equation] and the gradient w.r
Revised: In the M-step , note that the normalized importance weights are constant, the proposal distribution is fixed, and the gradient w.r.t.

Original: [Equation]

Original: t. L SI-NP ( ϑ ; η k , ϑ k ) can be estimated in a straightforward way as follows.
Revised: L SI-NP ( ϑ ; η k , ϑ k ) can be estimated in a straightforward way as follows.

Original: [Equation]
Revised: [Equation]

Original: We can see the role of normalized importance weights in gradient estimates from Eq.
Revised: We can see the role of normalized importance weights in gradient estimates from Eq.

Original: (33): remember that
Revised: (33): remember that

Original: [Equation]
Revised: [Equation]

Original: ; ϑ k ) in the k -th iteration when the functional prior works as the proposal distribution, so the more gradients will be allocated to those particles with higher generative likelihoods.
Revised: ϑ k ) in the k -th iteration when the functional prior works as the proposal distribution, so the more gradients will be allocated to those particles with higher generative likelihoods.

Original: Meanwhile, the normalized importance weights influence the optimization of the functional prior distribution so that the functional prior can match the best set of particles well.
Revised: Meanwhile, the normalized importance weights influence the optimization of the functional prior distribution so that the functional prior can match the best set of particles well.

Original: F C ONNECTION WITH CNP S F. P RIOR C OLLAPSE IN SI-NP S WITH O NE M ONTE C ARLO S AMPLE
Revised: F C ONNECTION WITH CNP S F. P RIOR C OLLAPSE IN SI-NP S WITH O NE M ONTE C ARLO S AMPLE

Original: Theorem 1 (L’Hˆopital’s Rule (Hospital, 1696)) Let f ( x ) and g ( x ) be two functions differentiable on an open interval I except possibly at a point c contained in I .
Revised: Theorem 1 (L’Hˆopital’s Rule (Hospital, 1696)) Let f ( x ) and g ( x ) be two functions differentiable on an open interval I except possibly at a point c contained in I .

Original: If lim x → c f ( x ) =
Revised: If lim x → c f ( x ) =

Original: lim x → c g ( x ) = ∞ , and (cid:16) 1 g ( x ) f ( x ) [Equation]
Revised: lim x → c g ( x ) = ∞ , and (cid:16) 1 g ( x ) f ( x ) (cid:17) ′ ̸

Original: ′ ̸ = 0 with ∀ x ∈ I and x ̸ = c , we can have the following limit equation.
Revised: = 0 with ∀ x ∈ I and x ̸ = c , we can have the following limit equation.

Original: [Equation]
Revised: [Equation]

Original: Proof F.1 (Proposition 3) Note that in our default setup of SI-NPs, the functional prior works as the proposal distribution to sample the latent variable.
Revised: Proof F.1 (Proposition 3) Note that in our default setup of SI-NPs, the functional prior works as the proposal distribution to sample the latent variable.

Original: Let z ∈ R d be the latent variable for a diagonal Gaussian conditional prior p ( z |D
Revised: Let z ∈ R d be the latent variable for a diagonal Gaussian conditional prior p ( z |D

Original: Cτ ; ϑ ) =
Revised: Cτ ; ϑ ) =

Original: N ( z ; µ ϑ ( D C τ ) , Σ ϑ ( D C τ )) .
Revised: N ( z ; µ ϑ ( D C τ ) , Σ ϑ ( D C τ )) .

Original: Here the learned mean and the covariance matrix are simply denoted by µ ϑ = [ µ 1 , . . .
Revised: Here the learned mean and the covariance matrix are simply denoted by µ ϑ = [ µ 1 , . . .

Original: , µ d ] T ∈ R d and Σ ϑ = diag (cid:2) σ 21 , . .
Revised: , µ d ] T ∈ R d and Σ ϑ = diag (cid:2) σ 21 , . .

Original: , σ 2 d (cid:3) .
Revised: , σ 2 d (cid:3) .

Original: With one Monte Carlo sample ˆ z = [ˆ z 1 , . .
Revised: With one Monte Carlo sample ˆ z = [ˆ z 1 , . .

Original: , ˆ z d ] T from the conditional prior, it can be written as ˆ z = µ ϑ + ˆ ϵ Σ 12 ϑ , ˆ ϵ ∼ N (0 , I d ) with help of a reparameterization trick (Kingma & Welling, 2013).
Revised: , ˆ z d ] T from the conditional prior, it can be written as ˆ z = µ ϑ + ˆ ϵ Σ 12 ϑ , ˆ ϵ ∼ N (0 , I d ) with help of a reparameterization trick (Kingma & Welling, 2013).

Original: [Equation]
Revised: [Equation]

Original: Eq.
Revised: Eq.

Original: (35) is the result of one Monte Carlo estimate, termed as the collapse term in the main paper.
Revised: (35) is the result of one Monte Carlo estimate, termed as the collapse term in the main paper.

Original: Built up on these, we rewrite the SI-NP optimization objective to maximize as Eq.
Revised: Built up on these, we rewrite the SI-NP optimization objective to maximize as Eq.

Original: [Equation]
Revised: [Equation]

Original: Now we prove that when the learned variance parameter { σ i } di =1 collapse into the value zero, the optimization objective in Eq.
Revised: Now we prove that when the learned variance parameter { σ i } di =1 collapse into the value zero, the optimization objective in Eq.

Original: (35) and Eq.
Revised: (35) and Eq.

Original: (36) can be maximized.
Revised: (36) can be maximized.

Original: To simplify the notation, we put the mean variable µ ϑ aside, focus more on the variance variable Σ ϑ and let the value µ i − ˆ z idenoted by κ i .
Revised: To simplify the notation, we put the mean variable µ ϑ aside, focus more on the variance variable Σ ϑ and let the value µ i − ˆ z idenoted by κ i .

Original: We can directly apply L’Hˆopital’s Rule in Theorem (1) to these quantities and obtain the following equations. 
Revised: We can directly apply L’Hˆopital’s Rule in Theorem (1) to these quantities and obtain the following equations. [Equation]

Original: [Equation]
Revised: [Equation]

Original: Putting them together, the Gaussian latent variable will finally collapse into a Dirac delta distribution and this demonstrates the equivalence between SI-NP with one Monte Carlo sample and
Revised: Putting them together, the Gaussian latent variable will finally collapse into a Dirac delta distribution and this demonstrates the equivalence between SI-NP with one Monte Carlo sample and

Original: CNP.
Revised: CNP.

Original: F.
Revised: F.

Original: E FFECT OF A DJUSTING C OEFFICIENTS OF THE P RIOR R EGULARIZER
Revised: E FFECT OF A DJUSTING C OEFFICIENTS OF THE P RIOR R EGULARIZER

Original: Remark 

Revised: Remark

Original: Either increasing the number of Monte Carlo samples or lower the weight of the collapse term in SI-NPs can effectively avoid the prior collapse.
Revised: Either increasing the number of Monte Carlo samples or lower the weight of the collapse term in SI-NPs can effectively avoid the prior collapse.

Original: With increase of Monte Carlo samples, we can see the scale of the generative term outweights that of the collapse term, which indicates more weights are put in the gradient w.r.t.
Revised: With increase of Monte Carlo samples, we can see the scale of the generative term outweights that of the collapse term, which indicates more weights are put in the gradient w.r.t.

Original: the variance parameters to maximize the generative log-likelihood.
Revised: the variance parameters to maximize the generative log-likelihood.

Original: In this way, it can naturally avoid the prior collapse caused by the second term.
Revised: In this way, it can naturally avoid the prior collapse caused by the second term.

Original: Next [Equation] we analyze the influence of the coefficients α with α ∈ (0 , 1) for the generative and functional prior likelihoods.
Revised: Next, we analyze the influence of the coefficients α with α ∈ (0 , 1) for the generative and functional prior likelihoods.

Original: To this end [Equation] we go back to the original meta learning objective and introduce the α -dependent one as L ( ϑ ; α ) in Eq.
Revised: To this end, we go back to the original meta learning objective and introduce the α -dependent one as L ( ϑ ; α ) in Eq.

Original: [Equation]
Revised: [Equation]

Original: As noticed in L ( ϑ ; α ) , the KL divergence constraint enforces the functional prior p ( z
Revised: As noticed in L ( ϑ ; α ) , the KL divergence constraint enforces the functional prior p ( z

Original: |D
Revised: |D

Original: Cτ ; ϑ ) to be close to the learned functional posterior p ( z |D Tτ ; ϑ k ) .
Revised: Cτ ; ϑ ) to be close to the learned functional posterior p ( z |D Tτ ; ϑ k ) .

Original: Since the functional posterior is derived from the full observation of a function, this suggests that the relation of conditional entropy H ( z |D Tτ ) ≤
Revised: Since the functional posterior is derived from the full observation of a function, this suggests that the relation of conditional entropy H ( z |D Tτ ) ≤

Original: H ( z |D Cτ ) .
Revised: H ( z |D Cτ ) .

Original: This trait is also empirically supported by Fig.
Revised: This trait is also empirically supported by Fig.

Original: (6), where the functional priors with fewer context points exhibit higher values in the trace of the covariance matrices.
Revised: (6), where the functional priors with fewer context points exhibit higher values in the trace of the covariance matrices.

Original: As consequence, larger α values impose less constraint on the KL divergence term and results in higher entropy of p ( z |D
Revised: As consequence, larger α values impose less constraint on the KL divergence term and results in higher entropy of p ( z |D

Original: Cτ ; ϑ ) .
Revised: Cτ ; ϑ ) .

Original: G E XPERIMENTAL S ETUP & I MPLEMENTATION D ETAILS
Revised: G E XPERIMENTAL S ETUP & I MPLEMENTATION D ETAILS

Original: In all experiments, we use Adam (Kingma & Welling, 2013) as the default optimizer for all experiments.
Revised: In all experiments, we use Adam (Kingma & Welling, 2013) as the default optimizer for all experiments.

Original: Pytorch 5 works as the toolkit to program and run experiments.
Revised: Pytorch 5 works as the toolkit to program and run experiments.

Original: The implementation of vanilla NPs/CNPs/ANPs follows the repository 6 and related work (Garnelo et al., 2018a;b; Kim et al., 2019) in DeepMind.
Revised: The implementation of vanilla NPs/CNPs/ANPs follows the repository 6 and related work (Garnelo et al., 2018a;b; Kim et al., 2019) in DeepMind.

Original: The implementation of ML-NPs is based on that in (Foong et al., 2020) except that the convolution modules are removed for the fair comparison since the inference objective is our research focus.
Revised: The implementation of ML-NPs is based on that in (Foong et al., 2020) except that the convolution modules are removed for the fair comparison since the inference objective is our research focus.

Original: To enable researchers to implement our developed method in studies, we leave the anonymous Github link here: https://anonymous.4open.
Revised: To enable researchers to implement our developed method in studies, we leave the anonymous Github link here: https://anonymous.4open.

Original: science/r/SI_NPs-C832 , where we provide an example of SI-NPs.
Revised: science/r/SI_NPs-C832 , where we provide an example of SI-NPs.

Original: The full code implementations of our method will be released in the final version.
Revised: The full code implementations of our method will be released in the final version.

Original: G. M ETA L EARNING D ATASETS
Revised: G. M ETA L EARNING D ATASETS

Original: Synthetic Regression.
Revised: Synthetic Regression.

Original: We use the Gaussian process simulator to generate different stochastic functions.
Revised: We use the Gaussian process simulator to generate different stochastic functions.

Original: Three types of kernels are used to formulate diverse Gaussian processes.
Revised: Three types of kernels are used to formulate diverse Gaussian processes.

Original: This is the same as that in (Lee et al., 2020; Kawano et al., 2020).
Revised: This is the same as that in (Lee et al., 2020; Kawano et al., 2020).

Original: For each task, we generate x from the uniform distribution U [ − 2 . 0 , 2 . 0] and the mean function is zero.
Revised: For each task, we generate x from the uniform distribution U [ − 2 . 0 , 2 . 0] and the mean function is zero.

Original: The number of context points is drawn from n ∼ U [3 , 47] and the number of target points is n + m with m ∼ U [3 , 50 − n ] .
Revised: The number of context points is drawn from n ∼ U [3 , 47] and the number of target points is n + m with m ∼ U [3 , 50 − n ] .

Original: The following kernel functions specify different Gaussian processes in this paper.
Revised: The following kernel functions specify different Gaussian processes in this paper.

Original: • Matern − 52 kernel:
Revised: • Matern − 52 kernel:

Original: [Equation]
Revised: [Equation]

Original:  with d = 4 | x − x ′ | , s ∼ U [0 . 1 , 1 . 0] and l ∼ U [0 . 1 , 0 . 6] ; • RBF kernel: 
Revised: x, x [Equation] with d = 4 | x − x ′ | , s ∼ U [0 . 1 , 1 . 0] and l ∼ U [0 . 1 , 0 . 6] ; • RBF kernel: k

Original: [Equation]
Revised: [Equation]

Revised: x,

Revised: [Equation]

Revised: [Equation]

Revised: =

Revised: [Equation]

Revised: exp (cid:18)

Revised: [Equation]

Original: with s ∼ U [0 . 1 , 1 . 0] and l ∼ U [0 . 1 , 0 . 6] • Periodic kernel:
Revised: with s ∼ U [0 . 1 , 1 . 0] and l ∼ U [0 . 1 , 0 . 6] • Periodic kernel:

Original: [Equation]
Revised: [Equation]

Original: with s ∼ U [0 . 1 , 1 . 0] , l ∼ U [0 . 1 , 0 . 6] and p ∼ U [0 . 1 , 0 . 5]
Revised: with s ∼ U [0 . 1 , 1 . 0] , l ∼ U [0 . 1 , 0 . 6] and p ∼ U [0 . 1 , 0 . 5]

Original: Image Datasets.
Revised: Image Datasets.

Original: Benchmark image datasets include MNIST (Bottou et al., 1994), FMNIST (Xiao et al., 2017), CIFAR10 (Krizhevsky et al., 2009) and SVHN (Sermanet et al., 2012).
Revised: Benchmark image datasets include MNIST (Bottou et al., 1994), FMNIST (Xiao et al., 2017), CIFAR10 (Krizhevsky et al., 2009) and SVHN (Sermanet et al., 2012).

Original: In meta training and testing, we randomly select the number of context pixels n for each sampled batch of images ( n ∼ U [1 , 784] in MNIST/FMNIST and n ∼ U [1 , 1023] in CIFAR10/SVHN).
Revised: In meta training and testing, we randomly select the number of context pixels n for each sampled batch of images ( n ∼ U [1 , 784] in MNIST/FMNIST and n ∼ U [1 , 1023] in CIFAR10/SVHN).

Original: For pixel values, they are transformed to normalized Tensors via pytorch package.
Revised: For pixel values, they are transformed to normalized Tensors via pytorch package.

Original: All other set-ups are the same as in (Garnelo et al., 2018a;b; Kim et al., 2019).
Revised: All other set-ups are the same as in (Garnelo et al., 2018a;b; Kim et al., 2019).

Original: Sim2Real Dataset.
Revised: Sim2Real Dataset.

Original: This is a part of additional experiments.
Revised: This is a part of additional experiments.

Original: We retain the preprocessing and meta training set-up in (Gordon et al., 2019).
Revised: We retain the preprocessing and meta training set-up in (Gordon et al., 2019).

Original: The meta training datasets include the Lotka-Volterra simulation samples and Predator-Prey’s real-world dataset.
Revised: The meta training datasets include the Lotka-Volterra simulation samples and Predator-Prey’s real-world dataset.

Original: All datasets are normalized before the meta training process.
Revised: All datasets are normalized before the meta training process.

Original: Please refer to (Gordon et al., 2019) for more details.
Revised: Please refer to (Gordon et al., 2019) for more details.

Original: G. N EURAL A RCHITECTURES & O PTIMIZATIONS & E VALUATION S ET - UP
Revised: G. N EURAL A RCHITECTURES & O PTIMIZATIONS & E VALUATION S ET - UP

Original: Synthetic Regression.
Revised: Synthetic Regression.

Original: In terms of neural architectures, we use the same setup as that in (Gordon et al., 2019; Lee et al., 2020) for all baselines.
Revised: In terms of neural architectures, we use the same setup as that in (Gordon et al., 2019; Lee et al., 2020) for all baselines.

Original: The dimension of latent variables is 128.
Revised: The dimension of latent variables is 128.

Original: The
Revised: The

Original: Encoder is a two hidden layer neural network with 128 neuron units for each layer.
Revised: Encoder is a two hidden layer neural network with 128 neuron units for each layer.

Original: The Decoder is a one hidden layer neural network with 128 neuron units.
Revised: The Decoder is a one hidden layer neural network with 128 neuron units.

Original: The optimizer’s learning rate is 5 e − 4 .
Revised: The optimizer’s learning rate is 5 e − 4 .

Original: For all methods, we sample 100 tasks as one batch to train in each iteration, and the number of iteration steps in meta training is 100000.
Revised: For all methods, we sample 100 tasks as one batch to train in each iteration, and the number of iteration steps in meta training is 100000.

Original: In meta training, the numbers of Monte Carlo samples for
Revised: In meta training, the numbers of Monte Carlo samples for

Original: ML-NPs and SI-NPs are 16, while those in meta testing are 32.
Revised: ML-NPs and SI-NPs are 16, while those in meta testing are 32.

Original: Image Completion.
Revised: Image Completion.

Original: The setup is the same with that in (Garnelo et al., 2018a) and works for all
Revised: The setup is the same with that in (Garnelo et al., 2018a) and works for all

Original: NPs variants.
Revised: NPs variants.

Original: As default, we set the dimension of latent variables z as 128 for all baselines.
Revised: As default, we set the dimension of latent variables z as 128 for all baselines.

Original: For all baselines, the Encoder is constituted with three hidden layers (128 neuron units each).
Revised: For all baselines, the Encoder is constituted with three hidden layers (128 neuron units each).

Original: The
Revised: The

Original: Decoder has five hidden layers (128 neuron units each) as well.
Revised: Decoder has five hidden layers (128 neuron units each) as well.

Original: The learning rate for the optimizer is 5 e − 4 .
Revised: The learning rate for the optimizer is 5 e − 4 .

Original: The training batch size for all images is 

Revised: The training batch size for all images is

Original: and we meta train the model until convergence (the maximum epoch number for MNIST/FMNIST is 100, and that for CIFAR10/SVHN is 200, and early stop is used when it reaches convergence).
Revised: and we meta train the model until convergence (the maximum epoch number for MNIST/FMNIST is 100, and that for CIFAR10/SVHN is 200, and early stop is used when it reaches convergence).

Original: In meta training, the numbers of Monte Carlo samples are 16 for ML-NPs and 8 for SI-NPs (We find that 8 Monte Carlo samples are enough for
Revised: In meta training, the numbers of Monte Carlo samples are 16 for ML-NPs and 8 for SI-NPs (We find that 8 Monte Carlo samples are enough for

Original: SI-NPs to obtain competitive performance and increasing the number of particles will require more computations but the performance improvement is minor).
Revised: SI-NPs to obtain competitive performance and increasing the number of particles will require more computations but the performance improvement is minor).

Original: In meta testing, the Monte Carlo sample numbers are 32 for latent variable models.
Revised: In meta testing, the Monte Carlo sample numbers are 32 for latent variable models.

Original: Sim2Real.
Revised: Sim2Real.

Original: We use the same setup as that in (Gordon et al., 2019; Lee et al., 2020) for all baselines.
Revised: We use the same setup as that in (Gordon et al., 2019; Lee et al., 2020) for all baselines.

Original: The neural architectures are the same as the Synthetic regression settings, except the dimension of output variables changes to 2.
Revised: The neural architectures are the same as the Synthetic regression settings, except the dimension of output variables changes to 2.

Original: For other settings, please refer to (Gordon et al., 2019) for more details.
Revised: For other settings, please refer to (Gordon et al., 2019) for more details.

Original: As in (Garnelo et al., 2018a), for the Decoder in all models, we use the modified standard deviation variable ˆ σ i = 0 .
Revised: As in (Garnelo et al., 2018a), for the Decoder in all models, we use the modified standard deviation variable ˆ σ i = 0 .

Original: 1 + 0 .
Revised: 1 + 0 .

Original: 9 ∗ σ i for the output distribution p ( y i | x i , z ; ϑ ) in all benchmarks, where σ i =
Revised: 9 ∗ σ i for the output distribution p ( y i | x i , z ; ϑ ) in all benchmarks, where σ i =

Original: MLP ϑ
Revised: MLP ϑ

Original: ( x i , z i ) .
Revised: ( x i , z i ) .

Original: Evaluation Set-up of Datasets.
Revised: Evaluation Set-up of Datasets.

Original: For Table (1)/(2) in meta testing, we keep the same set-up as (Foong et al., 2020; Gordon et al., 2019; Lee et al., 2020;
Revised: For Table (1)/(2) in meta testing, we keep the same set-up as (Foong et al., 2020; Gordon et al., 2019; Lee et al., 2020;

Original: Kawano et al., 2020), which randomly select the number of context points in each batch and average the testing results of all batches as the final result.
Revised: Kawano et al., 2020), which randomly select the number of context points in each batch and average the testing results of all batches as the final result.

Original: The range of the number of context points can be found in the above subsection.
Revised: The range of the number of context points can be found in the above subsection.

Original: H M ORE E XPERIMENTAL R ESULTS H. SI-NP S WITH A L EARNABLE P ROPOSAL D ISTRIBUTION
Revised: H M ORE E XPERIMENTAL R ESULTS H. SI-NP S WITH A L EARNABLE P ROPOSAL D ISTRIBUTION

Original: We have investigated the use of a learnable proposal distribution in SI-NPs.
Revised: We have investigated the use of a learnable proposal distribution in SI-NPs.

Original: In this case, an additional proposal distribution q η ( z ) =
Revised: In this case, an additional proposal distribution q η ( z ) =

Original: N ( z ; µ η ( D Tτ ) , Σ η ( D Tτ )) is introduced to sample the latent variables.
Revised: N ( z ; µ η ( D Tτ ) , Σ η ( D Tτ )) is introduced to sample the latent variables.

Original: The neural architecture of such a proposal distribution is the same as the approximate posterior in vanilla NPs, but the role is entirely distinguished.
Revised: The neural architecture of such a proposal distribution is the same as the approximate posterior in vanilla NPs, but the role is entirely distinguished.

Original: Unfortunately, we find that it is difficult to stabilize performance even
Revised: Unfortunately, we find that it is difficult to stabilize performance even

Original: though coefficients of the generative log-likelihood ln p ( D Tτ | z ( b ) ; ϑ ) , the functional prior log-likelihood ln p ( z ( b ) |D
Revised: though coefficients of the generative log-likelihood ln p ( D Tτ | z ( b ) ; ϑ ) , the functional prior log-likelihood ln p ( z ( b ) |D

Original: Cτ ; ϑ ) and the proposal likelihood ln q η ( z ( b ) |D Tτ ) are tuned in a lot of trials.
Revised: Cτ ; ϑ ) and the proposal likelihood ln q η ( z ( b ) |D Tτ ) are tuned in a lot of trials.

Original: Since there is no convergence solution when assigning the same weights to minimize Eq.
Revised: Since there is no convergence solution when assigning the same weights to minimize Eq.

Original: (31) and the negative of Eq.
Revised: (31) and the negative of Eq.

Original: (12) with a shared optimizer, we do not report the result here.
Revised: (12) with a shared optimizer, we do not report the result here.

Original: H. E VALUATION WITH M ORE M ONTE C ARLO P ARTICLES
Revised: H. E VALUATION WITH M ORE M ONTE C ARLO P ARTICLES

Original: In the main paper, evaluating the exact likelihood L ( ϑ ) = ln (cid:2)(cid:82) p ( D Tτ | z ; ϑ ) p ( z |D
Revised: In the main paper, evaluating the exact likelihood L ( ϑ ) = ln (cid:2)(cid:82) p ( D Tτ | z ; ϑ ) p ( z |D

Original: Cτ ; ϑ ) dz (cid:3) is intractable for the studied latent variable models, so we report the evaluation results by setting B the number of the Monte Carlo particles in Eq.
Revised: Cτ ; ϑ ) dz (cid:3) is intractable for the studied latent variable models, so we report the evaluation results by setting B the number of the Monte Carlo particles in Eq.

Original: (39) fixed.
Revised: (39) fixed.

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Empirical Explanation.
Revised: Empirical Explanation.

Original: We set B = 32 in evaluation based on the empirical observations because the evaluated likelihood for all experiments nearly reaches the convergence or does not significantly increase.
Revised: We set B = 32 in evaluation based on the empirical observations because the evaluated likelihood for all experiments nearly reaches the convergence or does not significantly increase.

Original: To see this point, we give the illustration of the evaluation results by varying the number of Monte Carlo samples and analyzing the test log-likelihoods in image completion tasks.
Revised: To see this point, we give the illustration of the evaluation results by varying the number of Monte Carlo samples and analyzing the test log-likelihoods in image completion tasks.

Original: As displayed in Fig.
Revised: As displayed in Fig.

Original: (8), SI-NPs show performance improvement with more particles in
Revised: (8), SI-NPs show performance improvement with more particles in

Original: FMNIST/CIFAR10/SVHN, and B = 32 is sufficient enough in evaluation 7 .
Revised: FMNIST/CIFAR10/SVHN, and B = 32 is sufficient enough in evaluation 7 .

Original: For CNPs, the performance does not change with more particles since the functional prior is collapsed.
Revised: For CNPs, the performance does not change with more particles since the functional prior is collapsed.

Original: While for NPs, the performance goes far worse than importance-weighted ones and does not significantly improve with more particles.
Revised: While for NPs, the performance goes far worse than importance-weighted ones and does not significantly improve with more particles.

Original: So we only show the ML-NPs and SI-NPs for better visual comparison.
Revised: So we only show the ML-NPs and SI-NPs for better visual comparison.

Original: Theoretical Explanation.
Revised: Theoretical Explanation.

Original: Note that ML-NPs and SI-NPs are importance weighted methods, the number of Monte Carlo samples matters.
Revised: Note that ML-NPs and SI-NPs are importance weighted methods, the number of Monte Carlo samples matters.

Original: It is theoretically proved in Theorem 1 in the paper importance weighted autoencoders (Burda et al., 2016) that L MC ( ϑ ; B 1 ) ≥ L MC ( ϑ ; B 2 ) with B 1 ≥ B 2 .
Revised: It is theoretically proved in Theorem 1 in the paper importance weighted autoencoders (Burda et al., 2016) that L MC ( ϑ ; B 1 ) ≥ L MC ( ϑ ; B 2 ) with B 1 ≥ B 2 .

Original: Our developed SI-NPs can be viewed as the conditional version of importance weighted autoencoders, which explains the empirical observations in Fig.
Revised: Our developed SI-NPs can be viewed as the conditional version of importance weighted autoencoders, which explains the empirical observations in Fig.

Original: However, when the prior is collapsed to a deterministic embedding, we do not expect this effect.
Revised: However, when the prior is collapsed to a deterministic embedding, we do not expect this effect.

Original: H. I NFLUENCE OF D IMENSIONS OF L ATENT V ARIABLES
Revised: H. I NFLUENCE OF D IMENSIONS OF L ATENT V ARIABLES

Original: It is unrealistic to explore all hyper-parameters’ influence: e.g., learning rates, numbers of layers, dimensions of each layer, types of activation function, batch size in training, dimensions of latent variables, the Cartesian of these hyper-parameters causes dimension explosion, and the required time of running experiments can be more than one year with limited GPUs.
Revised: It is unrealistic to explore all hyper-parameters’ influence: e.g., learning rates, numbers of layers, dimensions of each layer, types of activation function, batch size in training, dimensions of latent variables, the Cartesian of these hyper-parameters causes dimension explosion, and the required time of running experiments can be more than one year with limited GPUs.

Original: So we keep most of the set-up of the above hyper-parameters the same as that in original papers of CNP/NP/ANPs.
Revised: So we keep most of the set-up of the above hyper-parameters the same as that in original papers of CNP/NP/ANPs.

Original: [Figure]
Revised: [Figure]

Original: In this subsection
Revised: In this subsection

Original: [Figure]
Revised: [Figure]

Original: we study the influence of dimensions of latent variables on SI-NPs.
Revised: we study the influence of dimensions of latent variables on SI-NPs.

Original: Considering that the meta training process is time expensive, we report the result on FMNIST image completion in Table (6).
Revised: Considering that the meta training process is time expensive, we report the result on FMNIST image completion in Table (6).

Original: It can be observed that this factor rarely influences test performance.
Revised: It can be observed that this factor rarely influences test performance.

Original: [Table]
Revised: [Table]

Original: H. S IM 2R EAL E XPERIMENTAL R ESULTS
Revised: H. S IM 2R EAL E XPERIMENTAL R ESULTS

Original: Following that in (Gordon et al., 2019; Lee et al., 2020), We conduct experiments in Lotka-Volterra dynamical systems.
Revised: Following that in (Gordon et al., 2019; Lee et al., 2020), We conduct experiments in Lotka-Volterra dynamical systems.

Original: The transition dataset is collected in the mentioned simulator for meta training.
Revised: The transition dataset is collected in the mentioned simulator for meta training.

Original: The testing scenarios include Lotka-Volterra simulators and a Predator-Prey’s real-world dataset
Revised: The testing scenarios include Lotka-Volterra simulators and a Predator-Prey’s real-world dataset

Original: Hudson’s Baye hare-lynx.
Revised: Hudson’s Baye hare-lynx.

Original: The meta testing results are reported in Table (7).
Revised: The meta testing results are reported in Table (7).

Original: It can be seen that SI-NPs can achieve the best performance in Lotka-Volterra and Predator-Prey datasets.
Revised: It can be seen that SI-NPs can achieve the best performance in Lotka-Volterra and Predator-Prey datasets.

Original: Since the Predator-Prey dataset is out of the meta training distribution, the log-likelihoods are significantly lower than the Lotka-Volterra ones. 
Revised: Since the Predator-Prey dataset is out of the meta training distribution, the log-likelihoods are significantly lower than the Lotka-Volterra ones. [Table] [Table] [Table]

Revised: [Table]

Original: P
Revised: [Table] [Table] [Table] [Table]

Original: [Table]
Revised: [Table]

Original: p
Revised: i o n

Original: [Table]
Revised: [Table]

Original: l a t i o n
Revised: PredatorContextTarget

Original: [Table]
Revised: [Table]

Original: P o p u l a t i o n
Revised: P o p u l a t i o n

Original: [Figure]
Revised: [Figure]

Original: Meanwhile, we plot the predicted predator and prey population evolution in Fig.
Revised: Meanwhile, we plot the predicted predator and prey population evolution in Fig.

Original: As illustrated, meta trained SI-NPs can roughly characterize the trends and they can capture critical turning points.
Revised: As illustrated, meta trained SI-NPs can roughly characterize the trends and they can capture critical turning points.

Original: [Table]
Revised: [Table]

Original: H. A UGMENTING SI-NP S WITH A TTENTION N ETWORKS
Revised: H. A UGMENTING SI-NP S WITH A TTENTION N ETWORKS

Original: Theoretically, we can combine different optimization objectives of NPs with various structural inductive biases.
Revised: Theoretically, we can combine different optimization objectives of NPs with various structural inductive biases.

Original: Take the attention inductive bias as an example; we augment the vanilla SI-NP with attention networks the same as in (Kim et al., 2019) and compare the augmented one with other augmented baselines.
Revised: Take the attention inductive bias as an example; we augment the vanilla SI-NP with attention networks the same as in (Kim et al., 2019) and compare the augmented one with other augmented baselines.

Original: To enable fair comparison, we also augment other baselines with attention networks.
Revised: To enable fair comparison, we also augment other baselines with attention networks.

Original: We apply the modification to all methods, and this operation results in ANP (Kim et al., 2019), ML-ANP and SI-ANPs.
Revised: We apply the modification to all methods, and this operation results in ANP (Kim et al., 2019), ML-ANP and SI-ANPs.

Original: Neural Architectures.
Revised: Neural Architectures.

Original: We retain all neural architectures the same as in Appendix (G.2), except for adding the deterministic path to obtain a local deterministic variable (Kim et al., 2019).
Revised: We retain all neural architectures the same as in Appendix (G.2), except for adding the deterministic path to obtain a local deterministic variable (Kim et al., 2019).

Original: The deterministic path is built with a cross attention encoder.
Revised: The deterministic path is built with a cross attention encoder.

Original: Due to memory restriction, one head is used to obtain a 128 dimensional local variable, and the neural network set-up for query/key/value is the same as (Kim et al., 2019) 8 .
Revised: Due to memory restriction, one head is used to obtain a 128 dimensional local variable, and the neural network set-up for query/key/value is the same as (Kim et al., 2019) 8 .

Original: Optimization.
Revised: Optimization.

Original: We retain the optimization step the same as those in Appendix (G.2).
Revised: We retain the optimization step the same as those in Appendix (G.2).

Original: H.5. S YNTHETIC R EGRESSION
Revised: H .5. S YNTHETIC R EGRESSION

Original: In Table (8)
Revised: In Table (8)

Original: [Table]
Revised: [Table]

Original: it can be seen SI-ANPs outperform ANPs in all kernel cases.
Revised: it can be seen SI-ANPs outperform ANPs in all kernel cases.

Original: SI-ANPs show a slight advantage over ML-ANPs in Marten and RBF kernels.
Revised: SI-ANPs show a slight advantage over ML-ANPs in Marten and RBF kernels.

Original: We also observe anomaly results of ANPs in the Periodic kernel case, which illustrates that the use of attention might deteriorate the performance.
Revised: We also observe anomaly results of ANPs in the Periodic kernel case, which illustrates that the use of attention might deteriorate the performance.

Original: [Table]
Revised: [Table]

Original: H.5. I MAGE C OMPLETION
Revised: H.5. I MAGE C OMPLETION

Original: In Table (9)
Revised: In Table (9)

Original: [Table]
Revised: [Table]

Original: we report the results in image completion.
Revised: we report the results in image completion.

Original: We notice that the SI-ANP significantly beats other models in FMNIST/SVHN/CIFAR10 and is comparable with the ML-ANP in MNIST.
Revised: We notice that the SI-ANP significantly beats other models in FMNIST/SVHN/CIFAR10 and is comparable with the ML-ANP in MNIST.

Original: Meanwhile, the performance gap between vanilla NPs and SI-NPs in Table (2) is quite huge
Revised: Meanwhile, the performance gap between vanilla NPs and SI-NPs in Table (2) is quite huge

Original: [Table]
Revised: [Table]

Original: while this situation is alleviated after adding attention modules.
Revised: while this situation is alleviated after adding attention modules.

Original: All of these indicate that incorporating structural inductive biases in complicated tasks is also necessary.
Revised: All of these indicate that incorporating structural inductive biases in complicated tasks is also necessary.

Original: Combining SI-NPs with more powerful structural inductive biases is the top choice for boosting performance.
Revised: Combining SI-NPs with more powerful structural inductive biases is the top choice for boosting performance.

Original: [Table]
Revised: [Table]

Revised: S IM 2R

Revised: [Table]

Original: In Table (10)
Revised: In Table (10)

Original: [Table]
Revised: [Table]

Original: we report the results with the attention module augmentation.
Revised: we report the results with the attention module augmentation.

Original: As observed, ANPs,
Revised: As observed, ANPs,

Original: ML-ANPs
Revised: ML-ANPs

Original: [Table]
Revised: [Table]

Original: and SI-ANPs exhibit comparable performance in Lotka-Volterra simulation.
Revised: and SI-ANPs exhibit comparable performance in Lotka-Volterra simulation.

Original: As for the Predatory-Prey testing results, the conclusion is similar to that in Table (7) without attention augmentations.
Revised: As for the Predatory-Prey testing results, the conclusion is similar to that in Table (7) without attention augmentations.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: In this case, we guess the local deterministic embedding from the attention network plays a more critical role than the global latent variable in fitting these two datasets.
Revised: In this case, we guess the local deterministic embedding from the attention network plays a more critical role than the global latent variable in fitting these two datasets.

Original: The visualization of fitting
Revised: The visualization of fitting

Original: Predator-Prey samples with SI-ANPs is given in Fig.
Revised: Predator-Prey samples with SI-ANPs is given in Fig.

Original: The real-world samples are well fitted with well quantified standard deviations.
Revised: The real-world samples are well fitted with well quantified standard deviations.

Original: We notice that the measured predictive mean square errors and the negative log-likelihoods of the target data points in the Predator-Prey testing dataset are comparable using SI-NPs and SI-ANPs.
Revised: We notice that the measured predictive mean square errors and the negative log-likelihoods of the target data points in the Predator-Prey testing dataset are comparable using SI-NPs and SI-ANPs.

Original: However, the fitting result of the context data points are distinguished a lot: the mean square error of SI-NPs in 4 runs is 0 .
Revised: However, the fitting result of the context data points are distinguished a lot: the mean square error of SI-NPs in 4 runs is 0 .

Original: 285 ± 0 .
Revised: 285 ± 0 .

Original: 007 and that of SI-ANPs is 0 .
Revised: 007 and that of SI-ANPs is 0 .

Original: 029 ± 0 .
Revised: 029 ± 0 .

Original: This implies that the attention network in the real-world dataset focuses more on the context point fitting while the global latent variable is for the general trend characterization.
Revised: This implies that the attention network in the real-world dataset focuses more on the context point fitting while the global latent variable is for the general trend characterization.

Original: P o p u l a t i o n
Revised: P o p u l a t i o n

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure] [Figure]
Revised: PredatorContextTarget 2.

Original: n
Revised: P

Original: [Figure]
Revised: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure] [Table] [Figure] [Table]
Revised: n

Original: [Figure]
Revised: [Figure]

Original: [Table][Figure] [Table] [Figure]
Revised: [Table]

Original: [Table]

Original: [Figure]
Revised: [Figure]

Original: H. A DDITIONAL V ISUALIZATIONS H.6. M ORE S YNTHETIC R EGRESSION R ELATED R ESULTS
Revised: H. A DDITIONAL V ISUALIZATIONS H.6. M ORE S YNTHETIC R EGRESSION R ELATED R ESULTS

Original: We include more visualized results with the meta-trained models for Gaussian process datasets in
Revised: We include more visualized results with the meta-trained models for Gaussian process datasets in

Original: Fig.
Revised: Fig.

Original: An illustrated, periodic kernel cases are most challenging, and SI-NPs can learn faint but crucial fluctuation signals from mean functions, while vanilla NPs fail to capture them.
Revised: An illustrated, periodic kernel cases are most challenging, and SI-NPs can learn faint but crucial fluctuation signals from mean functions, while vanilla NPs fail to capture them.

Original: For other cases, the behaviors of these models are similar in characterizing trends: vanilla NPs tend to show higher variance in Marten kernel cases.
Revised: For other cases, the behaviors of these models are similar in characterizing trends: vanilla NPs tend to show higher variance in Marten kernel cases.

Original: SI-NPs and ML-NPs are comparable in Marten and RBF kernel cases.
Revised: SI-NPs and ML-NPs are comparable in Marten and RBF kernel cases.

Original: CNPs seem to best match the context points in both cases.
Revised: CNPs seem to best match the context points in both cases.

Original: In Fig.s (12)/(13)/(14), we respectively plot the curve fitting results in all kernel cases when all latent variable models are augmented by attention networks.
Revised: In Fig.s (12)/(13)/(14), we respectively plot the curve fitting results in all kernel cases when all latent variable models are augmented by attention networks.

Original: In Marten and RBF cases, all attention augmented models well fit data points.
Revised: In Marten and RBF cases, all attention augmented models well fit data points.

Original: Nevertheless, in RBF cases, SI-NPs can better capture data point-dependent deviations.
Revised: Nevertheless, in RBF cases, SI-NPs can better capture data point-dependent deviations.

Original: In Periodic cases, it can be observed that SI-ANPs can precisely capture fluctuations and quantify more convincing uncertainty.
Revised: In Periodic cases, it can be observed that SI-ANPs can precisely capture fluctuations and quantify more convincing uncertainty.

Original: Sometimes, ANPs underestimate the uncertainty while ML-ANPs fail to show data point distinguished uncertainty
Revised: Sometimes, ANPs underestimate the uncertainty while ML-ANPs fail to show data point distinguished uncertainty

Original: . H.6. M ORE I MAGE C OMPLETION R ELATED R ESULTS
Revised: . H.6. M ORE I MAGE C OMPLETION R ELATED R ESULTS

Original: In Fig.
Revised: In Fig.

Original: (15), the scale of computed KL divergence values in vanilla NPs is positively correlated with the semantic complexity.
Revised: (15), the scale of computed KL divergence values in vanilla NPs is positively correlated with the semantic complexity.

Original: Also, note that the approximate functional prior in vanilla NPs seldom collapses, but it has a theoretical bias away from the optimal functional prior according to Remark (1).
Revised: Also, note that the approximate functional prior in vanilla NPs seldom collapses, but it has a theoretical bias away from the optimal functional prior according to Remark (1).

Original: We can also find with more context points, the approximate prior is closer to the approximate posterior, so the value decreases accordingly.
Revised: We can also find with more context points, the approximate prior is closer to the approximate posterior, so the value decreases accordingly.

Original: In Fig.
Revised: In Fig.

Original: (16), we sample a collection of images from MNIST/FMNIST/SVHN/CIFAR10 to visualize the completed results.
Revised: (16), we sample a collection of images from MNIST/FMNIST/SVHN/CIFAR10 to visualize the completed results.

Original: We can find that without structural inductive biases, SI-NPs can reasonably complete the images in MNIST/FMNIST/SVHN and exhibit the uncertainty from partial observations.
Revised: We can find that without structural inductive biases, SI-NPs can reasonably complete the images in MNIST/FMNIST/SVHN and exhibit the uncertainty from partial observations.

Original: The learned functional prior, such as that in SVHN, can generate images that are different from
Revised: The learned functional prior, such as that in SVHN, can generate images that are different from

Original:  [Figure]
Revised: [Table] [Figure] [Table] [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Original: [Figure] 
Revised: [Figure] [Table] [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: in Periodic Kernel Cases.
Revised: Periodic Kernel Cases

Original: [Figure]
Revised: [Figure]

Original: In meta testing, we still vary the number of context points in image datasets.
Revised: In meta testing, we still vary the number of context points in image datasets.

Original: For NPs, the KL divergence value 
Revised: For NPs, the KL divergence value D

Original: [Equation]
Revised: [Equation]

Revised: is computed.

Original: the ground truth due to partial observation.
Revised: the ground truth due to partial observation.

Original: As for CIFAR10, it has more complicated semantics and is challenging for SI-NPs with only MLPs in neural architectures.
Revised: As for CIFAR10, it has more complicated semantics and is challenging for SI-NPs with only MLPs in neural architectures.

Original: For cases when SI-NPs are augmented by attention neural networks (Kim et al., 2019), we show more generated examples with learned SI-ANPs in Fig.
Revised: For cases when SI-NPs are augmented by attention neural networks (Kim et al., 2019), we show more generated examples with learned SI-ANPs in Fig.

Original: As can be seen, the completed results are not blurred and have a high quality.
Revised: As can be seen, the completed results are not blurred and have a high quality.

Original: Notably, the quantified variances are decreased with the increase in the context points, which shows excellent asymptotic behavior in this domain.
Revised: Notably, the quantified variances are decreased with the increase in the context points, which shows excellent asymptotic behavior in this domain.

Original: I C OMPUTATION T OOLS
Revised: I C OMPUTATION T OOLS

Original: In this project we use NVIDIA 1080-TiGPUs to finish all experiments.
Revised: In this project we use NVIDIA 1080-TiGPUs to finish all experiments.