Neural Processes with Stochastic Attention: Paying More Attention to the Context Dataset
Abstract
Neural processes (NPs) aim to stochastically complete unseen data points based on a given context dataset.
NPs essentially leverage a given dataset as a context representation to derive a suitable identiﬁer for a novel task.
To improve the prediction accuracy, many variants of NPs have investigated context embedding approaches that generally design novel network architectures and aggregation functions satisfying permutation invariant.
In this work, we propose a stochastic attention mechanism for NPs to capture appropriate context information.
From the perspective of information theory, we demonstrate that the proposed method encourages context embedding to be differentiated from a target dataset, allowing
NPs to consider features in a target dataset and context embedding independently.
We observe that the proposed method can appropriately capture context embedding even under noisy data sets and restricted task distributions, where typical NPs suffer from a lack of context embeddings.
We empirically show that our approach substantially outperforms conventional NPs in various domains through 1D regression, predator-prey model, and image completion.
Moreover, the proposed method is also validated by MovieLens-10k dataset, a real-world problem.
Introduction
Neural processes (NPs) have been in the spotlight as they stochastically complete unseen target points considering a given context dataset without huge inference computation [1, 2, 3].
NPs leverage neural networks to derive an identiﬁer suitable for a novel task using context representation, which contains information about given context data points.
These methods enable us to handle considerable amounts of data points, such as in image-based applications, that the Gaussian process cannot naively deal with.
Many studies have revealed that the prediction performance relies on the way of context representation [3, 4].
The variants of NPs have mainly investigated on context embedding approaches that generally design novel network architectures and aggregation functions that are permutation invariant.
For example, the standard neural processes is developed using MLP layers to embed context set via mean aggregation [1, 2].
Bayesian aggregation satisfying permutation invariance enhanced the prediction accuracy by varying weights for individual context data points [4].
From the prospective of network architectures to increase model capacity, Kim et al.
(2019) suggested the way of local representation to consider relations between context dataset and target dataset using deterministic attention mechanism [3, 5].
For robustness under noisy situations, bootstrapping method was proposed to orthogonally apply to variants of NPs [6].
Despite many appealing approaches, one signiﬁcant drawback of previous NPs is that they still underﬁt when confronted with noisy situations like real-world problems.
This manifests as inaccurate predictions at the locations of the context set as seen in Figure 1a.
Additionally, in Figure 1b, the attentive neural process (ANP) fails to capture contextual embeddings because the attention weights
[Figure]
all points lowest with is slightly improved
To address this issue, we propose a newly designed neural process to fundamentally improve performance by paying more attention to the context dataset.
The proposed method expedites a stochastic attention to adequately capture the dependency of the context and target datasets by adjusting stochasticity.
We observe that our proposed algorithm properly performs by utilizing contextual information in an intended manner as seen in Figure 1.
Thus, this paper clariﬁes the proposed method as regularized NPs in terms of information theory, explaining that the stochastic attention encourages the context embedding to be differentiated from the target dataset.
This differentiated information induces NPs to become appropriate identiﬁers for target dataset by paying more attention to context representation.
To summarize, we make the following contributions:
• We propose the novel neural process that pay more attention to the context dataset.
We claims for the ﬁrst time, using the information theory framework, that critical conditions for contextual embeddings in NPs are independent of target features and close to context datasets.
• Through comprehensive analyses, we illustrate how stochastic local embeddings are crucial for NPs to focus on capturing the dependencies of context and target datasets.
Even when context dataset contains noise or is somewhat different from the train dataset, as shown in the bootstrapping NP [6], the proposed method is capable of adapting a novel task while preserving predictive performance.
Particularly, this method signiﬁcantly enhances performance without additional architectures and data augmentation compared to the attentive neural process [3].
• The experimental results show that the proposed model substantially outperforms conventional NPs in typical meta-regression problems.
For instance, the proposed method achieves to obtain the state of the art score in the image completion task with the CelebA dataset.
Especially, the proposed model maintains performance in the limited task distribution regimes such as the MovieLenz-10k dataset with a small number of users.
Background 2.1 Neural Processes
Suppose that we have an observation set X
= { x i } ni =1 and a label set Y =
{ y i } ni =1 .
NPs [1, 2, 3] are designed to obtain the probabilistic mapping from the observation set to the label set p ( Y | X, X c , Y c ) given a small subset ( X c , Y c ) =
( x j , y j ) mj
=1 .
Basically, it is built upon the neural network with an encoder-decoder architecture where the encoder f φ outputs a task representation by feed-forwarding ( X c , Y c ) through permutation-invariant set encoding [7, 8] and the decoder f θ models the distribution of Y (e.g Gaussian case : estimating µ, σ ) using X along with the encoder outputs.
Its objective is to maximize the log likelihood over the (unknown) task distribution p ( T ) .
All tasks provided by the data generating process are considered as Monte Carlo samples, respectively:
[Equation]
where ( µ i , σ i ) =
f θ ( x i , r ) ,
[Equation]
The set encoding architecture differentiates the type of NPs.
In the conditional neural process (CNP) [1], the representation r is a deterministic variable such that, by using mean aggregation, the encoder maps the context set into a single deterministic representation r = 1 / m (cid:80) mj
=1 f φ ( x j , y j ) .
The neural process (NP) [2] introduces a probabilistic latent variable z to model functional uncertainty as a stochastic process such that the parameters of output distribution may change according to the sampled value of z .
Due to the intractable log-likelihood, a training objective is derived based on variational inference which can be decomposed into two terms, reconstruction term and regularization term:
[Equation]
For simplicity, the prior distribution p θ ( z | X c , Y c ) is approximated by q φ ( z | X c , Y c ) .
However, as pointed out in the ANP [3], the mean aggregation over the context set is too restrictive to describe the dependencies of the set elements.
To enhance the expressiveness of the task representation, the ANP accommodates an attention mechanism [5] into the encoder, which generates the local deterministic representation r i corresponding to a target data point x i , and addresses the underﬁtting issue in NPs.
Although additional set encoding methods such as the kernel method and the bayesian aggregation considering task information have been suggested [9, 10, 11, 4], the attentive neural process is mainly considered as the baseline in terms of model versatility.
2.2 Bayesian Attention Module
Consider m key-value pairs, packed into a key matrix K ∈ R m × d k and a value matrix V ∈ R m × d v and n queries packed into Q ∈ R n × d k , where the dimensions of the queries and keys are the same.
Attention mechanisms aim to create the appropriate values O ∈ R n × d v corresponding to Q based on the similarity metric to K , which are typically computed via an alignment score function g such that Φ = g ( Q, K ) .
Then, a softmax function is applied to allow the attention weight W ∈ R n × m to satisfy the simplex constraint so that the output features can be obtained by O i,j = W i
[Equation]
j :
[Equation] [Equation]
Note that there are many options for the alignment score function g where a scaled dot-product or a neural network are widely used.
The bayesian attention module [12] considers a stochastic attention weight W .
Compared to other stochastic attention methods [13, 14, 15, 16], it requires minimal modiﬁcation to the deterministic attention mechanism described above so that it is compatible with the existing frameworks, which can be adopted in a straightforward manner.
Speciﬁcally, un-normalized attention weights ˆ W such that W i,j = ˆ W i,j / (cid:80) mj
=
ˆ W i,j are sampled via the variational distribution q φ ( ˆ W | Q, K ) , which can be trained via amortized variational inference:
[Equation]
Considering the random variable ˆ W to be non-negative such that it can satisfy the simplex constraint by normalization, the variational distribution q φ ( ˆ W | Q, K ) is set to Weibull ( k, λ ) and the prior distribution p θ ( ˆ W ) is set to Gamma ( α, β ) .
This λ can be obtained by the standard attention mechanism, on the other hands, α can be either a learnable parameter or a hyper-parameter depending on that the model follows key-based contextual prior described in the following paragraph.
The remaining variables k and β are regarded as user-deﬁned hyper-parameters.
By introducing Euler–Mascheroni constant γ [17, 18], the KL divergence in Equation 4 can be computed in an analytical expression.
[Equation]
Samples from Weibull distribution can be obtained using a reparameterization trick exploiting an inverse CDF method: λ ( − log(1 − (cid:15) ))
1 k , (cid:15) ∼ Uniform (0 , 1) .
Note that mean and variance of
Weibull ( k, λ ) are computed as λ Γ(1 + 1 / k ) and λ 2 (cid:2) Γ(
+ 2 / k ) − (Γ(1 + 1 / k )) 2 (cid:3) .
It can be observed that the variance of obtained samples decreases as k increases.
Key-based contextual prior To model the prior distribution of attention weights, key-based contextual prior was proposed.
This method allows the neural network to calculate the shape parameter α of the gamma distribution as a prior distribution.
This leads to the stabilization of KL divergence between standard attention weights and sampled attention weights, and it prevents overﬁtting of the attention weights [12].
In this paper, we explain the reason why the key-based context prior is important for capturing an appropriate representation focusing on a context dataset from the prosepective of information theory.
Neural Processes with Stochastic Attention
We expect NPs to appropriately represent the context dataset to predict target data points in a novel task.
However, if the context dataset has a limited task distribution and noise like real-world problems, conventional NPs tend to sensitively react to this noise and maximize the objective function including irreducible noise.
These irreducible noises do not completely correlate with context information, so that this phenomenon derives meaningless set encoding of the context dataset in training phase and hinders adaptation to new tasks.
In other words, the output of NPs p θ ( y i | x i , z, r i ) does not depend on ( z,
r i ) .
To preserve the quality of context representation, we propose a method that better utilizes the context dataset by exploiting stochastic attention with the key-based contextual prior to NPs.
We show that the proposed method enables to create more precise context encoding by adjusting stochasticity, even in very restricted task distribution regimes.
3.1 Generative Process
As with the attentive neural process [3], the proposed method consists of the two types of encoders and a single decoder architecture.
The ﬁrst encoder embeds ( X c , Y c ) to a global representation z and the second encoder makes a local representation r i by ( x i , X c , Y c ) .
The global representation z serves to represent entire context data points ( X c , Y c ) , whereas the local representation r i is in charge of ﬁne-grained information between target data x
i and context ( X c , Y c ) for predicting the output distribution p ( y i | x i , ( z, r i )) .
Unlike the ANP, the proposed model considers all intermediate representations ( z, r i ) as stochastic variables.
We exploit the Bayesian attention module to create a local representation r i by the stochastic attention weights w i with the key-based contextual prior [12].
To use the reparameterization trick for sampling stochastic variables, we draw random noise samples (cid:15) 1 ∼ Unif (0 , 1) , (cid:15) 2 ∼ N (0 , 1) .
First, as mentioned in section 2, the stochastic attention weights { w i } Ni =1 are obtained via the inverse CDF of the Weibull distribution with random noise (cid:15) 1 .
For amortized variational inference, the prior distribution q φ ( { w } Ni =1 | K ( X c )) is also derived with the context dataset X c for implementing key-based contextual prior.
Meanwhile, (cid:15) 2 is used to obtain the global representation z sampled from the normal distribution.
The entire scheme is shown in Figure D.2c.
As shown in section 2, we can compute all the KL divergences of z and { w } ni =1 as closed-form solutions.
The decoder follows the standard neural processes p ( y i | x i , z, r i ) =
N ( y i | µ φ ( x i , z, r i ) , σ 2 φ ( x i , z, r i )) .
We provides Appendix D for implementation details.
3.2 Learning and Inference
Unlike the attentive neural process [3], the proposed method regards all representations z and
W = { w i } ni =1 as stochastic variables, so that we clearly drive the objective function according to amortized variational inference.
Based on the objective function of neural process [2], the proposed method adds KL divergence of stochastic attention weight W = { w i } ni =1 .
Assuming the independence between z and { w i } ni such that individual w i is only dependent on x i and ( X c , Y c ) as seen in Figure D.2c, the objective function for each task T k is presented as follows:
[Equation]
Note that ( X, Y ) follows a task T k and
each task T k is drawn from task distribution p ( T ) .
The ﬁnal objective function is L ( φ, θ ) =
E T k
[ L T k ( φ, θ )] .
From the perspective of amortized variational inference, the prior distributions of p ( z ) and p ( { w i } ) ni =1 should be deﬁned.
With regard to z , we follow the standard neural processes wherein p ( z ) is deﬁned as q φ ( z |{ X c , Y c } [2].
In the case of { w i } Ni =1 , we introduce the strategy of Bayesian attention modules [12].
The prior distribution of { w i } Ni =1 deﬁned as a key-based contextual prior, q φ ( w i | X c ) , not only stabilizes the KL divergence but also activates the representation w i to pay more attention to the context dataset.
In the next section, this objective function can be described in terms of information theory as role of regularization to pursue the original goal of NPs, which is deriving appropriate identiﬁers for target datasets in novel tasks.
3.3 Discussion of Objective Function from Perspectives of Information Theory
A novel part of the proposed method is the use of the stochastic attention mechanism to more leverage context information, thereby stably capturing the dependency of context and target datasets in noisy and very limited task distributions.
In this subsection, based on information theory, we elaborate the goal of NPs and our novel stochastic attention mechanism explained as the regularization of latent variables to pay more attention to context datasets.
This enables us to understand semantically latent variables and their requirements.
According to subsection 3.2, the objective function is categorized into two terms: a reconstruction log p θ ( y i | x i , z, r i ) and regularization as two KL Divergences.
Suppose that { x i , y i } is a data point in a target dataset, D is a given context dataset, latent variables z and r i are considered as Z , which is deﬁned as the information bottleneck for D , we suggest that maximizing a reconstruction term corresponds to increase I ( y i , D| x i ) and minimizing two KL
Divergence means to decrease I ( Z, x i | D ) .
First, the mutual information of the target value y i and the context dataset D given the input feature x i , I ( y i , D| x i ) , is the metric to identify that NPs is adapted to a novel task.
Suppose p ( y i | x i , D ) is output value of NPs and p ( y i | x i ) =
E D (cid:2) p ( y i | x i , D ) (cid:3) is output value not conditioned on context dataset D .
If NPs completely fail to adapt new tasks using the context dataset D , the suggested metric
I ( y i , D| x i ) goes to 0 .
[Equation]
Assumed that target data point { x i , y i } is sampled from T k , which is uncontrollable data generating process, the reconstruction term (cid:80) { x i ,y i } log p ( y i | x i , Z ) can be regarded to I ( y i ; Z | x i ) based on information bottleneck theorem; I ( y i ; D| x i ) ≥ I ( y i ; Z | x i ) holds.
A detailed explanation is given in
Appendix C.
The objective function should be designed to increase I ( y i , D| x i ) by an appropriate identiﬁer for a novel task considering a context dataset D .
However, as mentioned in the previous sections, NPs suffer from irreducible noises and limited task distribution in meta-training.
This phenomena means that NPs increase the objective function by learning the way to directly map from x i to y i including noises and some features only in meta-training, instead of taking into consideration of the context D .
To make latent variables Z capture proper dependencies, we deﬁne the regularization accounting for latent variables to pay more attention to the context dataset.
This regularization is the mutual information of latent variable Z and input x i given context dataset
D to indicates how similar information latent variable Z and target x i contain.
We regard p ( Z | x i , D ) as the distribution of latent variables that considers both context dataset D and target x i , meanwhile p ( Z | D ) =
E x i (cid:2) p ( Z | x i , D ) (cid:3) as the distribution of latent variables only depending on the context dataset D .
If latent variables Z have totally different information against the target x i given the context dataset D , It can be I ( Z, x i
[Equation]
=
[Equation]
We bring I ( y i ; D| x i ) − I
( Z, x i |D ) to be maximized instead of
I
( y i , D| x i ) so as to model adequately dependency between context and target datasets.
By decreasing I ( Z, x i |D ) and increasing I ( y i , D| x i ) in meta-training, we expect that the model learns the way to differentiate Z and x i and construct an identiﬁer which can further consider Z and x i together.
Note that φ is the parameter of the function to transform D to the information bottleneck Z , and θ is the parameter of the identiﬁer for target data point y i .
We demonstrate that I ( y i ; D| x i ) − I
( Z, x i |D ) ≥ I φ,θ ( y i ;
Z | x i ) − I φ ( Z, x i |D ) if φ , θ completely leverage all information about Z and x i .
We assume that neural network architectures can perform an intended manners.
Theorem 1.
Let Z be the representation of the context dataset D and it follows an information bottleneck.
The following equation holds when the latent variable Z can be split into { z, w i } and z is only dependent on D , but w i is dependent on D and x i ,
[Equation]
where w i is obtained by the Bayesian attention modules [12], and L T k is deﬁned in Equation 6 and its probabilistic graphical model follows Figure D.2c.
As mentioned, if the neural network architectures φ, θ are able to entirely represent to Z and y i given D and x i , L T k
( φ, θ )
=
I φ,θ ( y i ; Z | x i ) − I φ ( Z ; x i |D ) ideally holds.
In this subsection, we reveal that stochastic attention mechanism satisfy on this regularization.
p ( Z | x i , D ) is regarded as q φ ( w i | x i , { X c , Y c } ) and p ( Z |D ) is the key-based contextual prior; q φ ( w i | K ( X c )) .
Therefore, the stochastic attention mechanism with key contextual prior is considered as this regularization in Equation 8.
By Theorem 1, the network parameters φ and θ are updated by the gradient of the objective function in Equation 6 semantically improves I ( y i ; D| x i ) −
I
( Z, x i |D ) .
Therefore, we present the experimental results in the next section.
The empirical results indicate that we can implement a model that can complete unseen data points in novel tasks by fully considering the context representations.
The detailed analysis is presented in Appendix G.
Experiment
In this section, we describe our experiments to answer three key questions: 1) Are existing regularization methods such as weight decay, importance-weighted ELBO as well as recent methods(Information dropout, Bootstrapping, and Bayesian aggregation) effective to properly create context representations ?
; 2) Can the proposed method reliably capture dependencies between target and context datasets even under noisy situations and limited task distributions ?; 3) Is it possible to improve performance via the appropriate representation of the dataset ?
We choose CNP, NP [1, 2], and ANP [3] as baselines, which are commonly used.
For all baselines, we employ weight decay and importance-weighted ELBO that requires several samples against one data point and aggregated by log (cid:80) i exp L k .
Recent regularization methods can be also considered.
Information dropout [19, 20], Bayesian aggregation [4] and Bootstrapping [6] are chosen.
Bayesian aggregation is used for CNP and NP according to the original paper, meanwhile Information dropout and Bootstrapping are used for the ANP to verify regularization of local representations.
As a metric to evaluate the model’s predictive performance, we use the likelihood of the target dataset, while the likelihood of the context dataset is considered as a metric to measure the extent to which models can represent a context dataset.
Since these metrics are proportional to model performance, a higher value indicates a better performance.
In these experiments, we show that the proposed method substantially outperforms various NPs and their regularization methods.
4.1 1D Regression under Noisy Situations
We tested them with samples from various kernel functions (Matern, Periodic inclduing RBF) to identify that models is capable of adaptatation to new tasks.
To consider noisy environments,
we
[Table]
artiﬁcially generate noises into the training dataset.
Our model maintains outstanding performance compared to other baselines.
In particular, when the test data comes from periodic kernel GP [Table] which is the most different from the RBF GP, the proposed model utilizes context datasets appropriately regardless of the test data sources, whereas other models do not properly use the context data.
As shown in Figure 1a
[Table]
the proposed method captures context points better than other methods, as only it method correctly captures the relationship even under noisy situations.
4.2 Predator-Prey Model and Image Completion
[Table]
We apply NPs to a domain shift problem called the predator-prey model, proposed in the ConvCNP [9].
In this experiment
[Table]
we train models with the simulated data and applied models to predict the real-world data set, Hudson’s bay
the hare-lynx.
Note that the hare-lynx dataset is very noisy, unlike the simulated data.
The detailed explanation of datasets is written in Appendix F.
We observe that the proposed method is superior to the other models in both simulated test set and real-world dataset.
In particular, all baselines drastically decrease the likelihood values due to noises during the test, while the proposed model preserves the performance because it is robust to noises.
See the left side of Table 2.
We report the additional experimental result with periodic noises and the graphical explanation is presented in Appendix F.
Second, we conduct the image completion task in which the models generate images under some given pixels.
The experiment setting follows the previous experiments [1, 2, 3, 6].
We indicate that the proposed method records the best score as seen in the right side of Table 2.
The best baseline records 3.53 with 500 context pixels in Table 2, whereas our method attains 2.653 of likelihood even with 50 context pixels and grows to 3.948 with 500 context pixels.
The completed images by ours and baselines are shown in Appendix I.
4.3 Movielens-100k Data
We demonstrate the robustness and effectiveness of our method using a real-world dataset, the MovieLenz dataset, which is commonly used in recommendation systems.
This setting has an expensive data collection process that restricts the number of users.
In this section, we report how well the proposed method can be generalized to novel tasks using limited tasks during the meta-training.
[Table]
The proposed model performs better than the other methods.
As mentioned in subsection 4.2, it captures the information from the context dataset, while other methods suffer from noise in the data and lack of users in metatraining.
The experiment result of comaprison with baselines is reported in Appendix H. To validate use of real applications, we compare with existing studies.
According to Movielens-100k benchmarks, the state-of-the-art
RMSE score is about 0.890 [22].
As seen in Table 3, the
ANP points the comparable result of 0.909, meanwhile the proposed model attains a promising result, 0.895 of the
RMSE value.
This experiment indicates that the proposed method can reliably adapt to new tasks even if it provides small histories, and we identify again that our model can properly work on noisy situations.
Related Works
The stochastic attention mechanism enables the capturing of complicated dependencies and regularizing weights based on the user’s prior knowledge.
However, such methods cannot utilize backpropagation because they do not consider the reparameterization trick to draw samples [13, 14, 15, 16].
Recently, the Bayesian attention module suggests that the attention weights are samples from the
Weibull distribution whose parameters can be reparameterized, so this method can be stable for training and maintaining good scalability [12].
Since the conditional neural process have been proposed [1, 2], several studies have been conducted to improve the neural processes in various aspects.
The attentive neural process modiﬁes the set encoder as an cross-attention mechanism to increase the performance of the predictability and interpretability [3].
Some studies investigate NPs for sequential data [24, 25].
The convolutional network can be used for set encoding to obtain translation-invariant predictions specialized in functional representations with kernel functions [9, 10].
Similar to this study, the Bayesian context aggregation suggests the importance of context information over tasks [4] and the Bootstrapping NPs orthogonally improves the predictive performance under the model-data mismatch [6].
Conclusion
In this work, we utilize a stochastic attention mechanism to capture the relationship between the context and target dataset and adjust stochasticity during the training phase for making the model insensitive to noises.
A Closed Form Solution for The KL Divergence Between Weibull and Gamma Distribution
The generalized gamma distribution contains both Weibull and gamma distribution as special cases.
We are concerned with the three-parameter version of generalized gamma distribution introduced in
Stacy [26].
Its parameters can be categorized into one scale parameter a and two shape parameters d and p .
Its probability density function is deﬁned for x ∈ [0 , ∞ ) and given by
[Equation]
where Γ( · ) is the gamma function, and parameters a, d, p > 0 .
For d = p , it corresponds to the
Weibull distribution, and if p = 1 , it becomes the gamma distribution.
Bauckhage [18] derives a closed form solution for the kullback-leibler divergence between two generalized gamma distribution as follow.
[Equation]
where f 1 , f 2 are the generalized gamma distributions, and ψ ( · ) is digamma function.
The Weiull distribution with scale parameter λ and shape parameter K coincides with the generalized gamma distribution a = λ and d, p = K .
For gamma distribution with scale parameter β and shape parameter α , it becomes the generalized gamma distribution a = 1 / β , d
= α and p = 1 .
The KL divergence between the Weibull distribution ( λ, k ) and the gamma distribution ( α, β ) amounts to
[Equation]
We introduce ψ (1) = − γ ≈ 0 .
5772 as Euler constant.
Finally, we obtain Equation
[Equation]
B ELBO Derivations
In this section, we derive the objective function in the manuscript of this paper.
Without loss of generality, target dataset { X, Y } =
{ x i , y i } Ni =1 and context dataset D =
( X c , Y c )
= { x i , y i } Mj =
such that N (cid:29) M .
Let the log-likelihood of target data points in a task
T k be log p
( Y | X, D ) .
We begin to maximizing the log-likelihood of target data points based on context dataset and representations.
We suppose that it follows the graphical model in Figure D.2c
[Equation]
Applying Jensen’s inequality to
[Equation]
From Z i = { z, w i } in Figure D.2c, we deﬁne z is dependent on context dataset D and w i is dependent on target x
i and context dataset D
[Equation]
We assume that p ( w i | x i , D ) ≈
q φ 12 ( w i | K ( X c )) by regularization of latent variables to pay attention on context dataset.
It also follows the strategy of the standard neural processes : posterior distribution of z is q φ 2 ( w i |{ X t , Y t } ) and prior distribution of z is q φ 2 ( w i |{ X c , Y c } ) .
Thereby, we show that
ELBO become the objective function of the proposed method in Equation 6.
[Equation]
C Proof of Theorem
We present that I ( y i ; D| x i ) has relation to the objective function of the proposed method.
Suppose that Z is information bottleneck for D and all variables follow the graphical models Figure D.2c.
As described in Equation 7, the mutual information
I ( y i , D| x i ) is to measure dependency of y i and context dataset D .
We should maximize this information to generalize the novel task.
For simplicity, we set all latent variables as information bottleneck Z for the context dataset D .
Note that
Z = ( { w i } ni =1 , z ) is information bottleneck corresponding to context representations in NPs, where
W = { w i , . . . , w n } is an attention weights and X, Y = { ( x 1 , y 1 ) , . . .
, ( x n , y n ) } is respectively data points in the target data set.
φ is regarded as the parameter of the function to transform D to the information bottleneck Z and θ is the parameter of the identiﬁer for target data point y i .
By information bottleneck theorem, Equation C.14 holds.
[Equation]
The target data point { x i , y i } is drawn by the data generating process T k ∼ p
( T ) , so that H ( y | x i ) can be regarded as a constant value due to uncontrollable factors.
As a result, − H φ,θ ( y | x i , Z ) is written as
[Equation]
We employ (cid:80) { x i ,y i } log p θ ( y i | x i , Z φ ) , which is unbiased estimate for − H φ,θ ( y i | x i , Z ) based on
Monte Carlo sampling.
Given that the concrete information bottleneck Z is given, we expect that the
Z allows us to generalize a novel task { x i , y i } by enhancing (cid:80) { x i ,y i } log p θ ( y i | x i , Z φ ) .
However, in practice, Z induced by neural networks is more likely to memorize the training dataset including irreducible noises when the number of tasks is insufﬁcient.
The conventional methods accomplished maximizing (cid:80) { x i ,y i } log p θ ( y i | x
i , Z φ ) by simply ﬁnding the relation between y i and x i only in the training dataset.
It means that the information in representation Z becomes increasingly identical to the information in x i in the meta training dataset.
It does not satisfy the condition of information bottleneck for the context dataset D .
To avoid this issue, we introduce I ( Z, x i |D ) as a regularization.
By reducing dependencies of Z and x i , we can make the information bottleneck Z focus on D .
It means that as p ( Z | x i , D ) and P ( Z |D ) get closer, the target input x i less inﬂuences Z , but D has a high correlation with Z .
In other words, minimizing I ( Z, x i |D ) is to make Z and target x i more independent given context dataset D .
It encourages latent variables
Z = (
w i , z ) to pay more attention to context dataset D
[Equation]
where, the q ( w i | x i , D ) follows the Weibull distribution and q ( w i |D ) follows the gamma distribution.
In this study, q ( w i | x i , D ) can be modeled as the stochastic attention weights and q ( w i |D ) can be modeled as the key-based contextual prior q φ ( w i | K ) .
We can factorize all latent variables because the attention weight w i does not have a dependency on data points except for the speciﬁc data point { x i , y i } .
We denote that w i has conditional independence over { w j } i (cid:54) =
j given x i .
For the global representation; z , to follow objective function of neural processes, we assume q ( z | x i , D ) =
q φ ( z | X, Y ) and q φ ( z |D ) =
q ( z |D ) .
As mentioned in section 3, we suppose that the global representation z follows normal distribution.
Therefore, we derive the regularization term
I ( Z, x t |D ) have relation to
KL Divergence terms in our loss function.
From this fact, we recognize KL Divergence terms in our objective function helps latent variables give attention on context dataset.
To summarize the mutual information between y i and D , and the newly designed regularization, we found that
[Equation]
[Equation]
Based on the graphical models in Figure D.2c and assumptions of representations z and w i , we identify that this equation has relation to our objective function L T k
[Equation]
Finally, we can derive Theorem 1 as below
[Equation]
From Equation C.19, the gradients of φ and θ with respect to L T k can be regarded as the direction of increasing I ( y i ; D| x i ) − I ( Z, x i |D ) , where, L T k is deﬁned as the target likelihood and KL divergence of z and { w i } ni =1 in a single task.
[Equation]
where X D is input features in the context dataset and q φ ( w i | X D ) follows key-based contextual prior as described in section 3.
D Implement Details
We referred to most of the architectures from the paper[3] and their released source code 1 .
The information dropout and importance weighted ELBO were respectively borrowed from these papers[19, 27] 23 .
The stochastic attention can be implemented based on Bayesian attention modules[12] 4 .
We migrated and revised all codes to meet our purpose.
In this chapter, we follow the notation of this paper [6].
D.1 (Attentive) Neural Process
MLP Encoder
We suppose that multi-layers modules(MLP) have the structure as Equation D.21, where l is the number of layers, d in is the dimension of input features, d hidden is the dimension of hidden units and d out is the dimension of outputs.
[Equation]
The variants of neural processes have two types of MLP encoders: The deterministic path is used in the conditional neural process [1], and the stochastic path is employed in the neural process [2], attentive neural process [3] and ours.
The deterministic path is to aggregate all hidden units by the
MLP encoder.
[Equation]
Instead, the stochastic path is to aggregate all hidden units and then feed-forward a single network to generate µ φ 1 and σ φ 1 .
We obtain the stochastic path r via the reparameterization trick.
[Equation]
Attention encoder We introduce cross-attention to describe the dependency of context and target dataset.
Let MHA be a multi-head attention [5] computed as follows :
[Equation]
Where, { d q , d k , d v } are respectively the dimensions of query, key and value components.
The
LayerNorm is the layer normalization in terms of heads.
MLP Decoder
The architecture of MLP decoder is similar to the stochastic encoder.
In case of
CNP [1] and NP [2], this decoder transforms all representation z and the input feature { x i } i ∈ T to target distribution, the normal distribution, { µ i , σ i } i ∈ T .
[Equation]
Meanwhile, in case of attentive neural process [3] and ours, the inputs of this decoder are the global representation z , local representation { r i } i ∈ T and the input feature { x i } i ∈ T .
[Equation]
(D.26)
The detailed information of each architecture is described in Table D.
[Table] [Table]
[Table] [Table]
In this paper, we set the dimension of all latent variables as 128, namely d h = 128 .
The number of heads in multi-head attention is 8, which is the same as the original paper of attentive neural processes [3].
For all models and all experiments, we use the Adam optimizer [28] with the learning rate 0 .
001 , and we set the number of update steps as 200000 .
For training and evalation, we used AMD Ryzen 2950X(16-cores), RAM 64GB and RTX2080Ti.
The 1D regressions has a batch size of 1 and a total of 160 batches.
The Sim2Real conﬁgures the batch size to be 10 and the total number of batches to be 16.
The MovieLenz-100k has a batch size of 1 and the number of batches to be 459.
The image completion task consists of 227 batches and its size is 4.
D.2 Model architecture
We graphically show the architecture of the proposed method in Figure D.1.
This model has two types of encoder as attentive neural process [3].
The encoder parameters φ consists of φ 1 , φ 2 .
The φ 1 is responsible for the local representation r i and the φ 2 is responsible for global representation z .
The encoder of global representation is same as neural process [2], however, the encoder of local representation is different from the standard cross attention [5].
After obtaining standard attention weight w standard , we introduce reparameterization trick for w i , which follows the Weiubll distribution [12].
The important thing is that the key conceptual prior can be made by MLP φ 13 ( { x j } j ∈ C ) .
The decoder is the same as the attentive neural process [3].
D.3 Algorithm
The proposed method requires hyper-parameters k and β for reparameterization of the Weibull distribution and KL divergence between the Weibull and the gamma distribution.
We conduct the grid searches for k to ﬁnd the best value.
We identify that the proposed method with k = 300 adequately captures the dependency and generates noises to avoid memorization for all experiments.
In the case of β , we follow the setting of the Bayesian attention module [12].
We suggest the entire procedure of our algorithm as follows:
D.4 Comparison of probabilistic graphical models for variants of NPs
We describe NPs with probabilistic graphical models to differentiate the variants of NPs.
The Neural process employs mean aggregate function and reparameterization trick to obtain a global context representation z [2].
This model follows Figure D.2a.
The Attentive neural process expedites the multi-head cross attention to obtain local representation r i with global context representation z .
We show that the graphical model for ANP is the middle of Figure D.2, all variables in the attention mechanism are regarded as the determinstic variable.
In this work, we design that all latent variables contain stochasticity and achieves by the reparameterization trick.
By Bayesian attention module, we present our graphical model as shown in
Figure D.2c
Input: Task distribution p ( T ) , the shape parameters of weibull distribution and gamma distribution : k
= 300 (Image completion task : k = 100 ), β = 1 , Stepsize : γ output: encoder parameters : φ = { φ 1 , φ 2 } , decoder parameters : θ
Initialization : φ and θ randomly; while not converged do
Sample tasks {T k } Kk =1 from p ( T ) for all T k (cid:51) {T } do
Sample context dataset ( X c , Y c ) = { x j , y j } Mj =1 and target dataset ( X t , Y t ) =
{ x i , y i } Ni =1 from the task T k
Sample random noises (cid:15) 1 ∼ Unif (0 , 1) and (cid:15) 2 ∼ N (0 , 1)
Local representation { r i } Ni =1 :
Q, K, V ← f φ 1 ( X t , ( X c , Y c ))
w standard ← Softmax ( QK T / √ d k ) (Reprameterization sampling of weibull distribution for W ) λ ← w standard ∗ Γ(1 + 1 / k )
W = { w 1 , · · · , w n
[Equation]
← λ ( − log(1 − (cid:15) 1 )) 1 / k
W = { w 1 , · · · , w n } ← { w i (cid:80) W i , · · · , w n (cid:80) W i } (Parameters of the prior distribution for W )
α ← f φ 1 ( K ) { r i } Ni =1 ← { w i V, · · · , w n V }
Global representation z : µ φ 1 , σ φ 1 ← Aggregator ( f φ 2 ( { x j , y j } Mj =1 ))
(Reprameterization sampling of normal distribution for z ) z = µ φ 2 + (cid:15) 2 ∗ µ φ
Decode p ( Y t | X t , z, { r i } Ni =1 ) : µ y , σ y ← ( f θ ( X t , z, { r i } Ni =1 ))
Evaluation loss :
[Equation]
end
[Equation]
end
[Figure]
E 1D regressions
For the synthetic 1D regression experiment, we set d x = 1 , d y = 1 .
The number of layers in encoder and decoder in all baselines is respectively l enc = { l pre = 3 , l post = 1 } = 4 and l dec = 3 .
We set the dimension of latent variable as d h = 128 .
When it comes to data generation process, the training data is generated from the Gaussian process with RBF kernel.
For each task, we randomly generate x ∼ Unif ( − 2 , 2) and then y is the function value by the Gaussian process with RBF kernel, k
( x, x (cid:48) ) = s 2 · exp( − (cid:107) x − x (cid:48) (cid:107) 2 / 2 l 2 ) .
To validate our model, we establish two types of datasets in the this 1D regression experiment.
First, we set the parameters of the RBF kernel as s = 3 and l = 3 .
Hence, as shown in Figure E.3a, all datasets are drawn from continuous and smooth functions.
Second, we consider the noisy situations in the ﬁrst scheme.
We intend that the suggested scheme represents the actual real-world situation.
Unlike the existing studies [1, 3, 6], we modify the RBF kernel function adding a high frequency periodic kernel function
[Equation]
Lee et al.
(2020) proposed a noisy situation by using random noises sampled by t-distribution; however, these noises often have exaggerated values so that the generated function does not have any tendency and seems to be entire noises.
On the other hand, the function generated by Gaussian processes with a high frequent periodic kernel is smooth but is satisﬁed with a random function every trial.
Thus, this function does not interfere with the smoothness of the RBF GP function and maintains the smoothness of all support ranges.
Therefore, we decide to use the dataset generated by the RBF GP function with periodic noises to synthetically test all baselines and the proposed method for the robustness of noises.
The sampled dataset is graphically presented in Figure E.3b.
[Figure]
To generate function values of the Gaussian process, the GPy library 5 provides various functions compatible for PyTorch 6 .
In the GPy, (cid:107) x − x (cid:48) (cid:107) 2 can be regarded as the pre-deﬁned vector { 1 , . .
. , freq } T { 1 , . .
. , freq } .
We set the parameters of the periodic kernel as freq = 30 , p = 2 π , s = 1 .
The generated functions for training datasets are shown in Figure E.3.
To test generalization to a novel task, we introduce other functions such as the Matern32 kernel GP and the periodic kernel GP as shown in Lee et al.
In this experiment, we expect the trained models to capture context points despite different functions.
Naturally, all models perform well on the
RBF GP function because the meta-training set is generated from RBF GP functions; instead, we test all baselines on other types of functions such as the Matern32 kernel GP and the Periodic kernel GP as well as the RBF kernel GP.
For the Matern32 kernel
[Equation]
( x
[Equation]
) and the periodic kernel is same as mentioned earlier.
The shared parameters of these three GPs are same as s = 3 , l = 3 .
The periodic kernel requires another parameter freq = 10 , which is the default value in the GPy.
To elaborate the meta-training framework, the minimum number of context points is 3 , and the minimum number of target points is 6 , including the context data points.
The maximum number of context points and target points is respectively 97 and 100 .
In the following subsection, we report experimental results, including clean and noisy situations
[Table]
E.1 Experiment Result on RBF GP Functions
We train all baselines on the RBF GP functions without noises.
We demonstrate experimental results in terms of predictability and context set encoding
[Table]
As shown in Table E.
[Table]
all models are capable of ﬁtting the RBF GP function; meanwhile, all models are degraded in cases of Matern kernel GP and Periodic GP.
Unlike all baseline models, of which performances drop substantially, the proposed method has relatively small degradation.
Particularly, the proposed method records that the likelihood value of context points at Matern kernel GP and
Periodic GP is respectively 1.170 and 0.680.
Meanwhile, the best performance among baselines is 1.077 and 0.379 by ANP with weight decay.
This result shows that the proposed method performs better than all baselines.
The graphical results are described in Figure E.4.
E.2 Experiment Result on RBF GP Functions with Noises
As mentioned early in the current section, we train all models on the RBF GP functions adding the random periodic noise.
Looking at the column of RBF kernel GP in Table 1, most of the baselines are degraded due to noisy situations.
However, the performances of all baselines are improved in Matern
GP and Periodic GP.
We guess that this phenomenon can be explained as mitigating memorization issues by injecting random noise to the target value [29].
We present how effectively random noises improve generalization performances for all baselines and our method in this experiment.
However, we can recognize that injecting random noises cannot be a fundamental solution.
As seen in Appendix G, we emphasize that capturing and appropriately utilizing context information when predicting target values is a fundamental solution to avoid memorization in a given situation.
The experimental result is described in Table 1 on manuscript and the detailed graphical explanation is suggested in Figure E.5.
All models improve performance in Matern and Periodic GP than Figure E.4.
The proposed method outperforms baselines and regularization methods.
Even this model achieves better performance than the result on RBF GP without noises.
F Predator Prey Model
We follow the experimental detail in this paper [6, 9].
This experiment is designed to evaluate all baselines capable of adapting new tasks, which have slightly different task distributions.
It is called "Sim2Real".
We assume that, in this situation, we easily obtain simulation data, but it requires a high expense to collect real-world data points.
We try to train all baselines on the dataset generated by simulations and test them on real data sets that are relatively small compared to the simulation data.
The way of simulated data is generated from the Lotka-Volterra model(LV model) [30].
Note that X is the number of predators, and Y is the number of prey at any time step in our simulation.
According to the explanation, one of the following four events should occur.
The following events require the parameter θ 1 = 0 .
01 , θ 2 = 0 .
5 , θ 3 = 1 , θ 4 = 0 .
A single predator is born according to rate θ 1 · X · Y , increasing X by one. 2.
A single predator dies according to rate θ 2 · X , decreasing X by one. 3.
A single prey is born according to rate θ 3 · Y , decreasing Y by one. 4.
A single prey dies(or is eaten)
according to rate θ 4 · X · Y , decreasing Y by one.
The initial X and Y are randomly chosen.
On the other hand, Hudson’s bay lynx-hare dataset follows a similar tendency to the LV model; however, it is oscillating time series because it has been collected in real-world records.
There were unexpected outliers, unexplained events, and noise.
For the detailed explanation, refer Gordon et al.
(2019)’s work [9].
All simulation codes are available on this URL 7 .
The generated simulation data can be graphically shown as the left side of Figure F.6.
In this experiment, we set d x = 1 , d y = 2 , and all remaining settings are the same as the 1D regression experiment except that the number of context points is at least 15 and the extra target points is also at least 15.
When training models, we use the dataset generated by the LV model and test on the lynx-hare dataset.
The experimental result is shown in Table 2, and the prediction performance can be graphically shown in Figure F.6.
We conduct additional experiments to validate whether the random periodic noise positively inﬂuences the performance.
As conducted in the 1D regression experiment, we utilize the periodic kernel GP function, which has very high frequency freq = 100 to add noises to the training dataset.
As shown in
Table F.
compared to Table 2, We recognize that adding random periodic noise empirically improves all baselines and our method.
Among models, our models perform best due to the full representation of context and target datasets.
The prediction results can be graphically shown in Figure F.6b
[Table]
[Figure]
G Anlaysis of Context Embeddings via Attention Weights in 1D Regressions and the Lynx-hare Dataset
The attention mechanism can explicitly display the distance between context and target data points via attention weights.
Especially, when the dimension of features is one, x ∈ R 1 , the distance between features is simply calculated and sorted, so allowing for straightforward comprehension when the attention weights are shown in heat-maps.
As a result, we present several heatmaps to compare ours with baselines in both clean and noisy 1D regression datasets.
The horizontal axis in these graphs represents the value of features in the context dataset, while the vertical axis represents the value of features in the target dataset.
The best pattern for this heat-map is diagonal because all feature values are arranged in ascending order.
Additionally, we will provide the simpliﬁed heat-map that take into account only target points with same value in the context dataset.
Due to the fact that the original version has one-hundred target points, labels and ticks are necessarily small.
We guess that readers may be unable to decipher the detailed information such as labels and attention scores.
This simpliﬁed version allows the readers to instantly grasp what we attempt to convey by exhibiting a more distinct diagonal pattern.
From Figure G.7 to Figure G.9, on the left side, we show a simpliﬁed version of the attention score; on the right side, we show the result for the entire target dataset.
Plus, we provide Sim2Real’s attention heat-maps in both clean and noisy environments.
As a result, our discovery appears to be consistent.
First, the heatmaps of the 1D regression problem without perioidc noises are shown in Figure G.7.
In this graph, all models including ours can accurately depict the similarity between the context and targete dataset.
Although the attention scores for each model vary, it is clear that the majority pattern of all models is diagonal.
Therefore, all models are trained using the clean dataset in the intended manner.
However, other phenomena are found in the noisy data.
We identify that the attentive neural process [3] fails to capture contextual embeddings because the attention weights of all target points highlight on the lowest value or the maximum value in the context dataset.
In the case of the Bootstrapping ANP [6] and ANP with information dropout, the quality of heat-map is slightly improved, but it still falls short of ours.
This indicates that the present NPs are unable to properly exploit context embeddings because the noisy situations impair the learning of the context embeddings during meta-training.
On the other hands, even in the noisy situation, the attention score in ours still appear clearly.
Second, we include heatmaps for the Sim2Real problem.
As seen in
Figure F.6, the context and target datapoints are much too many for the label information and ticks to be recognized.
Hence, we recommend that readers verify the presence of diagonal patterns rather than examining detailed numerical values.
As seen in Figure G.9a, all models are capable of capturing properly similarity between the context and target datasets.
However, as with 1D regression, the diagonal pattern of the baselines is disrupted as seen in the left graph of Figure G.9b; nevertheless, ours retains the ideal pattern in both clean and noisy situations.
When comparing our model’s heat-map pattern in clean and noisy environments, there is a noteworthy point.
Our model is capable of learning adaptively how to focus on certain context datapoints depending on the extent of dirty data in the meta-train dataset.
The model is trained on the clean dataset to take into consideration nearby points as well as the corresponding point in the context dataset, hence, the heat-map gradually changes.
This is because the clean dataset has smooth values, y i,j , near a certain feature x i,j .
Meanwhile, in the noisy dataset, the model is trained to focus exclusively on corresponding points to the context dataset.
Hence, the heat-map in Fig E.5 (b) indicates that attention score of target datapoints that includes the context dataset has a high value, whilst the remainder points treat all context datapoints uniformly.
This phenomena occurs because there is less correlation between adjacent features x i,j and its labels
y i,j in the noisy situation during the meta-training.
[Figure]
H Movielenz-100k data
We follow the experimental detail in the paper [31].
The way to split into train, test, validation dataset is described in this work 8 .
Brieﬂy, all ratings are ﬁrst divided into two datasets, which respectively have user IDs and are not duplicated each other.
We deﬁne that one is the meta-training dataset and the other is the meta-test dataset.
In this experiment, we evaluate baselines that generalize to a new customer with small history data points.
To explain a training regime, the number of context is at least three and the objective function is computed with the remaining data points.
All users do not have the same amount of ratings so that the number of data points is varying during meta-training.
When testing baselines, we sample a few ratings from unobserved users and then evaluate the likelihood values for the remaining points.
To exclude users’ unique information, we drop the User ID from the set of features and only use general features such as age, occupation.
Lastly, in this experiment, we set d x = 44 , d y = 1 , and all remaining settings are the same as the 1D regression experiment.
The detailed explanation is in Galashov et al.
(2019)’s work [31].
The likelihood values on the
MovieLens-10k is reported in Table H.6.
[Table]
To compare RMSE scores with existing methods for Movielens-100k data, we suggest evaluating
NPs using u1.base and u1.test.
As mentioned, the prediction by NPs is based on given context data points.
First, we randomly choose a customer in the u1.test.
Second, we randomly sample a small number of ratings from u1.base, which is used as the meta-training data.
In this experiment, we decided to draw ten samples from u1.base and employ these points as the context data set.
The ten ratings is relatively a short history compared to the size of the u1.base.
Third, we perform predictions for the data points in the u1.test based on the context data points.
The important thing is that context and target data points are sampled from the same user but the different data set.
Plus, we do not touch in information in the test set, at least.
Based on the relationship between sampled context points and target points, the performance of NPs is somewhat variable, so we report ﬁve runs and average these results.
The experimental result is in Table 3.
I Image Completion
In the image completion task, we follow the training regime suggested by NPs [1, 2].
In this regime, d x means the pixel location and d y means the color RGB data.
Namely, we set d x = 2 and d y = 3 .
For high capacity representation, the dimension of all hidden units is determined as d h = 512 .
All remaining settings, including layer sizes, are the same as the 1D regression experiments.
When it comes to image completion tasks, according to these papers, NPs are known to complete facial images given partial pixels.
Although a few pixels are provided, NPs are capable of understanding the common appearance of a human face.
These models generate eye, mouth, and nose even if only a small number of local information is given, which is generally hard to recognize.
The Conditional neural process properly performs; however, it often loses its context representations.
The Attentive neural process tackles this issue to employ an attention mechanism.
Instead of the encoders consisting of MLP layers, the self-attention can improve the quality of pixels generated at the context data points.
However, as mentioned on subsection 4.2, scores of generated pixels in unseen regions are still problems.
Even though many pixels are provided, the quality of completed images is not improved.
In this work, the proposed method concentrates on given contextual pixels and complete images based on that information.
Hence, the proposed method completes an image with high uncertainty when few pixels are given.
Plus, it can be seen that the higher the pixels information, the higher the quality of generated images.
The numerical result is presented in Table 2, the graphical evidences are shown in from Figure I.10 to Figure I.13.
The proposed method allows for completion of images using only partially provided pixels, so it enables to complete larget images than the image size in the training dataset.
We produce images of various sizes using the proposed method, which is trained on the CelebA dataset with size of 32 × 32 .
The completed images are presented in Figure I.14.
When we qualitatively evaluate the completion results, the proposed method properly perform for generating large images without sacriﬁcing performance.
All completed images are quite close to the ground truth.
As a result, we conclude that the proposed method can be used for task adaptation since it can complete images that the model have never encountered during meta-training and whose sizes differ from the training dataset.
In particular, while the percentage of contributed pixels to the total number of pixels remains constant, the quality of completed images are enhanced according to the amount of provided pixels.
We understand that the number of pixels is more signiﬁcant than the ratio of context points.
This is because the model has a sufﬁcient pixel counts to work with when images are large.
For instance, employing only 300 pixels as a context dataset is insufﬁcient to complete the images.
Nonetheless, the completed images with 300 pixels are comparable to those with higher pixel counts and is of higher quality than the baselines in Figure I.12.
There is an interesting discovery of uncertainty values in the image completion task.
Basically, we expect that many pixels reduce the uncertainty of generated images.
However, the existing methods stably generate facial images, but uncertainty values do not decrease even if available pixels increase.
The proposed method makes high variance for a completed image under the small number of context points.
As shown in Figure I.15, it can decrease the uncertainty of completed images when the
[Figure]
number of context points is large.
From this fact, we identify that the proposed method performs in an intended manner.