N EURAL P ROCESSES WITH S TOCHASTIC A TTENTION : P AYING MORE ATTENTION TO THE CONTEXT DATASET
Anonymous authors
Paper under double-blind review
A BSTRACT
Neural processes (NPs) aim to stochastically complete unseen data points based on a given context dataset.
NPs essentially leverage a given dataset as a context representation to derive a suitable identiﬁer for a novel task.
To improve the prediction accuracy, many variants of NPs have investigated context embedding approaches that generally design novel network architectures and aggregation functions satisfying permutation invariant.
In this work, we propose a stochastic attention mechanism for NPs to capture appropriate context information.
From the perspective of information theory, we demonstrate that the proposed method encourages context embedding to be differentiated from a target dataset, allowing
NPs to consider features in a target dataset and context embedding independently.
We observe that the proposed method can appropriately capture context embedding even under noisy data sets and restricted task distributions, where typical NPs suffer from a lack of context embeddings.
We empirically show that our approach substantially outperforms conventional NPs in various domains through 1D regression, predator-prey model, and image completion.
Moreover, the proposed method is also validated by MovieLens-10k dataset, a real-world problem.
I NTRODUCTION
Neural processes (NPs) have been in the spotlight as they stochastically complete unseen target points considering a given context dataset without huge inference computation (Garnelo et al., 2018a;b;
Kim et al., 2019).
NPs leverage neural networks to derive an identiﬁer suitable for a novel task using context representation, which contains information about given context data points.
These methods enable us to handle considerable amounts of data points, such as in image-based applications, that the
Gaussian process cannot naively deal with.
Many studies have revealed that the prediction performance relies on the way of context representation (Kim et al., 2019; Volpp et al., 2021).
The variants of NPs have mainly investigated on context embedding approaches that generally design novel network architectures and aggregation functions that are permutation invariant.
For example, Garnelo et al.
(2018a;b) developed NPs using MLP layers to embed context set via mean aggregation.
Bayesian aggregation satisfying permutation invariance enhanced the prediction accuracy by varying weights for individual context data points (Volpp et al., 2021).
From the prospective of network architectures to increase model capacity, Kim et al.
(2019) suggested the way of local representation to consider relations between context dataset and target dataset using deterministic attention mechanism (Vaswani et al., 2017).
For robustness under noisy situations, bootstrapping method was proposed to orthogonally apply to variants of NPs (Lee et al., 2020).
Despite many appealing approaches, one signiﬁcant drawback of previous NPs is that they still underﬁt when confronted with noisy situations like real-world problems.
This manifests as inaccurate predictions at the locations of the context set as seen in Figure 1a.
Additionally, in Figure 1b, the attentive neural process Kim et al.
(2019) fails to capture contextual embeddings because the attention weights of all target points highlight on the lowest value or the maximum value in the context dataset.
In the case of the Bootstrapping ANP Lee et al.
and ANP with information dropout, the quality of heat-map is slightly improved, but it still falls short of ours.
This indicates that the present NPs are unable to properly exploit context embeddings because the noisy situations impair the learning of the context embeddings during meta-training.
[Figure]
The detailed analysis is shown in Appendix G
To address this issue, we propose a newly designed neural process to fundamentally improve performance by paying more attention to the context dataset.
The proposed method expedites a stochastic attention to adequately capture the dependency of the context and target datasets by adjusting stochasticity.
It results in improving prediction performance due to maintained context information.
We observe that our proposed algorithm works well by utilizing contextual information in an intended manner as seen in Figure 1.
This method outperforms current NPs and their regularization methods in all experiment settings.
Thus, this paper clariﬁes the proposed method as regularized NPs in terms of information theory, explaining that the stochastic attention encourages the context embedding to be differentiated from the target dataset.
This differentiated information induces NPs to become appropriate identiﬁers for target dataset by paying more attention to context representation.
To summarize, we make the following contributions:
• We propose the novel neural process that pay more attention to the context dataset.
We claims for the ﬁrst time, using the information theory framework, that critical conditions for contextual embeddings in NPs are independent of target features and close to contextual datasets.
• Through comprehensive analyses, we illustrate how stochastic local embeddings are crucial for NPs to focus on capturing the dependencies of context and target datasets.
Even when context dataset contains noise or is somewhat different from target dataset, as shown in Lee et al.
(2020), the proposed method is capable of adapting a novel task while preserving predictive performance.
Particularly, this method signiﬁcantly enhances performance without additional architectures and data augmentation compared to the attentive neural process (Kim et al., 2019).
• The experimental results show that the proposed model substantially outperforms conventional NPs in typical meta-regression problems.
For instance, the proposed method achieves to obtain the state of the art score in the image completion task with the CelebA dataset.
Especially, the proposed model maintains performance in the limited task distribution regimes such as the MovieLenz-10k dataset with a small number of users.
This paper is organized as follows.
We introduce background knowledge in section 2.
sectionpresents a neural process with stochastic attention and related works illustrated in section 5.
Experimental results are shown in section 4, concluding with section 6.
B ACKGROUND 2. N EURAL P ROCESSES
Suppose that we have an observation set X
= { x i } ni =1 and a label set Y =
{ y i } ni =1 .
NPs (Garnelo et al., 2018a;b; Kim et al., 2019) are designed to obtain the probabilistic mapping from the observation set to the label set p ( Y | X, X c , Y c ) given a small subset ( X c , Y c ) =
( x j , y j ) mj
=1 .
Basically, it is built upon the neural network with an encoder-decoder architecture where the encoder f φ outputs a task representation by feed-forwarding ( X c , Y c ) through permutation-invariant set encoding (Zaheer et al., 2017; Edwards & Storkey, 2017) and the decoder f θ models the distribution of Y (e.g Gaussian case : estimating µ, σ ) using X along with the encoder outputs.
Its objective is to maximize the log likelihood over the (unknown) task distribution p ( T ) .
All tasks provided by the data generating process are considered as Monte Carlo samples, respectively:
[Equation]
where ( µ i , σ i ) =
f θ ( x i , r ) ,
[Equation]
=
[Equation] [Equation]
x
[Equation]
The set encoding architecture differentiates the type of NPs.
In Conditional neural process(CNP) (Garnelo et al., 2018a), the representation r is a deterministic variable such that, by using mean aggregation, the encoder maps the context set into a single deterministic representation r = 1 / m (cid:80) mj
=1 f φ ( x j , y j ) .
Neural process (NP) (Garnelo et al., 2018b) introduces a probabilistic latent variable z to model functional uncertainty as a stochastic process such that the parameters of output distribution may change according to the sampled value of z .
Due to the intractable log-likelihood, a training objective is derived based on variational inference which can be decomposed into two terms, reconstruction term and regularization term:
[Equation]
For simplicity, the prior distribution p θ ( z | X c , Y c ) is approximated by q φ ( z | X c , Y c ) .
However, as pointed out in Kim et al.
(2019), the mean aggregation over the context set is too restrictive to describe the dependencies of the set elements.
To enhance the expressiveness of the task representation,
Attentive neural process(ANP) accommodates an attention mechanism (Vaswani et al., 2017) into the encoder, which generates the local deterministic representation r i corresponding to a target data point x i , and addresses the underﬁtting issue in NP.
Although additional set encoding methods such as the kernel method and the bayesian aggregation considering task information have been suggested (Xu et al., 2020; Volpp et al., 2021), the attentive neural process is mainly considered as the baseline in terms of model versatility.
2. B AYESIAN A TTENTION M ODULE
Consider m key-value pairs, packed into a key matrix K ∈ R m × d k and a value matrix V ∈ R m × d v and n queries packed into Q ∈ R n × d k , where the dimensions of the queries and keys are the same.
Attention mechanisms aim to create the appropriate values O ∈ R n × d v corresponding to Q based on the similarity metric to K , which are typically computed via an alignment score function g such that Φ = g ( Q, K ) .
Then, a softmax function is applied to allow the attention weight W ∈ R n × m to satisfy the simplex constraint so that the output features can be obtained by O i,j = W i · V j :
[Equation]
Note that there are many options for the alignment score function g where a scaled dot-product or a neural network are widely used.
Bayesian attention module (Fan et al., 2020) considers a stochastic attention weight W .
Compared to other stochastic attention methods (Shankar & Sarawagi, 2018; Lawson et al., 2018; Bahuleyan et al., 2018;
Deng et al., 2018), it requires minimal modiﬁcation to the deterministic attention mechanism described above so that it is compatible with the existing frameworks, which can be adopted in a straightforward manner.
Speciﬁcally, un-normalized attention weights ˆ W such that
W i,j = ˆ W i,j / (cid:80) mj
=
ˆ W i,j are sampled via the variational distribution q φ ( ˆ W | Q, K ) , which can be trained via amortized variational inference:
[Equation]
[Equation]
Considering the random variable ˆ W to be non-negative such that it can satisfy the simplex constraint by normalization, the variational distribution q φ ( ˆ W | Q, K ) is set to Weibull ( k, λ ) and the prior distribution p θ ( ˆ W ) is set to Gamma ( α, β ) .
This λ can be obtained by the standard attention mechanism, on the other hands, α can be either a learnable parameter or a hyper-parameter depending on that the model follows key-based contextual prior described in the following paragraph.
The remaining variables k and β are regarded as user-deﬁned hyper-parameters.
By introducing Euler–Mascheroni constant γ (Zhang et al., 2018; Bauckhage, 2014), the KL divergence in Equation 4 can be computed in an analytical expression (shown in Appendix A).
[Equation]
Samples from Weibull distribution can be obtained using a reparameterization trick exploiting an inverse CDF method: λ ( − log(1 − (cid:15) ))
1 k , (cid:15) ∼ Uniform (0 , 1) .
Note that mean and variance of
Weibull ( k, λ ) are computed as λ Γ(1 + 1 / k ) and λ 2 (cid:2) Γ(
+ 2 / k ) − (Γ(1 + 1 / k )) 2 (cid:3) .
It can be observed that the variance of obtained samples decreases as k increases.
Key-based contextual prior To model the prior distribution of attention weights, key-based contextual prior was proposed.
This method allows the neural network to calculate the shape parameter α of the gamma distribution as a prior distribution.
This leads to the stabilization of KL divergence between standard attention weights and sampled attention weights, and it prevents overﬁtting of the attention weights (Fan et al., 2020).
In this paper, we explain the reason why the key-based context prior is important for capturing an appropriate representation focusing on a context dataset from the prosepective of information theory.
N EURAL P ROCESSES WITH S TOCHASTIC A TTENTION
We expect NPs to appropriately represent the context dataset to predict target data points in a novel task.
However, if the context dataset has a limited task distribution and noise like real-world problems, conventional NPs tend to sensitively react to this noise and maximize the objective function including irreducible noise.
These irreducible noises do not completely correlate with context information, so that this phenomenon derives meaningless set encoding of the context dataset in training phase and hinders adaptation to new tasks.
In other words, the output of NPs p θ ( y i | x i , z, r i ) does not depend on ( z, r i ) .
To preserve the quality of context representation, we propose a method that better utilizes the context dataset by exploiting stochastic attention with the key-based contextual prior to NPs.
We show that the proposed method enables to create more precise context encoding by adjusting stochasticity, even in very restricted task distribution regimes.
3. G ENERATIVE P ROCESS
As with the Attentive neural process(Kim et al., 2019), the proposed method consists of the two types of encoders and a single decoder architecture.
The ﬁrst encoder embeds ( X c , Y c ) to a global representation z and the second encoder makes a local representation r i by ( x i , X c , Y c ) .
The global representation z serves to represent entire context data points ( X c , Y c ) , whereas the local representation r i is in charge of ﬁne-grained information between target data x
i and context ( X c , Y c ) for predicting the output distribution p ( y i | x i , ( z, r i )) .
Unlike the ANP, the proposed model considers all intermediate representations ( z, r i ) as stochastic variables.
We exploit the Bayesian attention module to create a local representation r i by the stochastic attention weights w i with the key-based contextual prior (Fan et al., 2020).
To use the reparameterization trick for sampling stochastic variables, we draw random noise samples (cid:15) 1 ∼ Unif (0 , 1) , (cid:15) 2 ∼ N (0 , 1) .
First, as mentioned in section 2, the stochastic attention weights
{ w i } Ni =1 are obtained via the inverse CDF of the Weibull distribution with random noise (cid:15) 1 .
For amortized variational inference, the prior distribution q φ ( { w } Ni =1 | K ( X c )) is also derived with context dataset X c for implementing key-based contextual prior.
Meanwhile, (cid:15) 2 is used to obtain the global representation z sampled from normal distribution.
The entire scheme is shown in Figure D.2c.
As shown in section 2, we can calculate all the KL divergences of z and { w } ni =1 as closed-form solutions.
The decoder follows the standard neural processes p ( y i | x i , z, r i ) =
N ( y i | µ φ ( x i , z, r i ) , σ 2 φ ( x i , z, r i )) .
See Appendix D for implementation details.
3. L EARNING AND I NFERENCE
Unlike the Attentive neural process(Kim et al., 2019), the proposed method regards all representations z and W = { w i } ni =1 as stochastic variables, so that we clearly drive the objective function according to amortized variational inference.
Based on the objective function of Neural process(Garnelo et al., 2018b), the proposed method adds KL divergence of stochastic attention weight W = { w i } ni =1 .
Assuming the independence between z and { w i } ni such that individual w i is only dependent on x i and ( X c , Y c ) as seen in Figure 2, the objective function for each task T k is presented as follows:
[Equation]
Note that ( X, Y ) follows a task T k and
each task T k is drawn from task distribution p ( T ) .
The ﬁnal objective function is L ( φ, θ ) =
E T k
[ L T k ( φ, θ )] .
From the perspective of amortized variational inference, the prior distributions of p ( z ) and p ( { w i } ) ni =1 should be deﬁned.
With regard to z , we follow the standard neural processes wherein p ( z ) is deﬁned as q φ ( z |{ X c , Y c } (Garnelo et al., 2018b).
In the case of { w i } Ni =1 , we introduce the strategy of Bayesian attention modules (Fan et al., 2020).
The prior distribution of { w i } Ni =1 deﬁned as a key-based contextual prior, q φ ( w i | X c ) , not only stabilizes the KL divergence but also activates the representation w i to pay more attention to the context dataset.
In the next section, this objective function can be described in terms of information theory as role of regularization to pursue the original goal of NPs, which is deriving appropriate identiﬁers for target datasets in novel tasks.
3. D ISCUSSION OF OBJECTIVE FUNCTION FROM VIEWS OF INFORMATION THEORY
A novel part of the proposed method is the use of the stochastic attention mechanism to more leverage context information, thereby stably capturing the dependency of context and target datasets in noisy and very limited task distributions.
In this subsection, based on information theory, we elaborate the goal of NPs and our novel stochastic attention mechanism explained as the regularization of latent variables to pay more attention to context datasets.
This enables us to understand latent variables and their requirements semantically.
According to subsection 3.2, the objective function is categorized into two terms: a reconstruction log p θ ( y i | x i , z, r i ) and regularization as two KL
Divergences.
Suppose that { x i , y i } is a data point in a target dataset, D is a given context dataset, latent variables z and r i are considered as Z , which is deﬁned as the information bottleneck for D , we suggest that maximizing a reconstruction term corresponds to increase I ( y i , D| x i ) and minimizing two KL Divergence means to decrease I ( Z, x i | D ) .
First, the mutual information of the target value y i and the context dataset D given the input feature x i , I ( y i , D| x i ) , is the metric to identify that NPs is adapted to a novel task.
Suppose p ( y i | x i , D ) is output value of NPs and p ( y i | x i ) =
E D (cid:2) p ( y i | x i , D ) (cid:3) is output value not conditioned on context dataset D .
If NPs completely fail to adapt new tasks using the context dataset D , the suggested metric
I ( y i , D| x i ) goes to 0 .
[Equation]
Assumed that target data point { x i , y i } is sampled from T k , which is uncontrollable data generating process, the reconstruction term (cid:80) { x i ,y i } log p ( y i | x i , Z ) can be regarded to I ( y i ; Z | x i ) based on information bottleneck theorem; I ( y i ; D| x i ) ≥ I ( y i ; Z | x i ) holds.
A detailed explanation is given in
Appendix C.
The objective function should be designed to increase I ( y i , D| x i ) by an appropriate identiﬁer for a novel task considering a context dataset D .
However, as mentioned in the previous sections, NPs suffer from irreducible noises and limited task distribution in meta training.
This phenomena means that NPs increase the objective function by learning the way to directly map from x i to y i including noises and some features only in meta-training, instead of taking into consideration of the context D .
To make latent variables Z capture proper dependencies, we deﬁne the regularization accounting for latent variables to pay more attention to the context dataset.
This regularization is the mutual information of latent variable Z and input x i given context dataset
D to indicates how similar information latent variable Z and target x i contain.
We regard p ( Z | x i , D ) as the distribution of latent variables that considers both context dataset D and target x i , meanwhile p ( Z | D ) =
E x i (cid:2) p ( Z | x i , D ) (cid:3) as the distribution of latent variables only depending on the context dataset D .
If latent variables Z have totally different information against the target x i given the context dataset D , It can be I ( Z, x i |D ) =
[Equation]
We bring I ( y i ; D| x i ) − I
( Z, x i |D ) to be maximized instead of
I
( y i , D| x i ) so as to model adequately dependency between context and the target dataset.
By decreasing I ( Z, x i |D ) and increasing
I ( y i , D| x i ) in meta-training
, we expect that the model learns the way to differentiate Z and x i and construct an identiﬁer which can further consider Z and x i together.
Note that φ is the parameter of the function to transform D to the information bottleneck Z and θ is the parameter of the identiﬁer for target data point y i , we show that I ( y i ; D| x i ) − I
( Z, x i |D ) ≥ I φ,θ ( y i ;
Z | x i ) − I φ ( Z, x i |D ) if φ , θ is perfectly able to leverage all information about Z and x i .
We assume that neural network architectures can perform an intended manners.
Theorem 1.
Let Z be the representation of the context dataset D and it follows an information bottleneck.
The following equation holds when the latent variable Z can be split into { z, w i } and z is only dependent on D , but w i is dependent on D and x i ,
[Equation]
where w i is obtained by the Bayesian attention modules (Fan et al., 2020), and L T k is deﬁned in
Equation 6 and its probabilistic graphical model follows Figure 2.
As mentioned, if the neural network architectures φ, θ are able to completely represent to Z and y i given D and x i , L T k
( φ, θ )
=
I φ,θ ( y i ; Z | x i ) − I φ ( Z ; x i |D ) ideally holds.
In this subsection, we reveal that stochastic attention mechanism satisfy on this regularization as mentioned.
p ( Z | x i , D ) is regarded as q φ ( w i | x i , { X c , Y c } ) and p ( Z |D ) is the key-based contextual prior q φ ( w i | K ( X c )) .
Therefore, the stochastic attention mechanism with key-prior that requires D and x i as well as being able to this regularization for latent variables to pay more attention to the context dataset in Equationis considered.
By Theorem 1, the network parameters φ and θ are updated by the gradient of the objective function in Equation 6 semantically improves I ( y i ; D| x i ) −
I
( Z, x i |D ) .
Therefore, we present the experimental results in the next section.
They indicate that we can implement a model that can complete unseen data points in novel tasks by fully considering the context representations.
The detailed analysis is presented in Appendix G and Appendix H.
E XPERIMENT
In this section, we describe our experiments to answer three key questions: 1) Are existing regularization methods such as weight decay, importance-weighted ELBO as well as recent methods(Information dropout, Bootstrapping, and Bayesian aggregation) effective to properly create context representations ?
; 2) Can the proposed method reliably capture dependencies between target and context datasets even under noisy situations and limited task distributions ?; 3) Is it possible to improve performance via the appropriate representation of the dataset ?
To demonstrate the need for neural process with stochastic attention, we choose CNP, NP (Garnelo et al., 2018a;b), and ANP (Kim et al., 2019) as baselines, which are commonly used.
For all baselines,
[Table]
we employ weight decay and importance-weighted ELBO that requires several samples against one data point and aggregated by log (cid:80)
i exp L k .
Recent regularization methods can be also considered.
Information dropout(Kingma et al., 2015; Achille & Soatto, 2018), Bayesian aggregation (Volpp et al., 2021) and Bootstrapping (Lee et al., 2020) are chosen.
Bayesian aggregation is used for CNP and NP according to the original paper, meanwhile Information dropout and Bootstrapping are used for the ANP to verify regularization of local representations.
When training, all methods follow their own learninig policy; however, for fair comparison, all methods perform one Monte-Carlo sampling across a test data point during testing.
We respectively conduct 5 trials for each experiment and report average scores.
As a metric to evaluate the model’s predictive performance, we use the likelihood of the target dataset, while the likelihood of the context dataset is considered as a metric to measure the extent to which models can represent a context dataset.
Since these metrics are proportional to model performance, a higher value indicates a better performance.
In these experiments, we show that the proposed method substantially outperforms various NPs and their regularization methods.
4. 1D REGRESSION UNDER NOISY SITUATIONS
We conduct synthetic 1D regression experiments to test whether our model adequately captures the dependency of a dataset under noisy situation.
We trained all models with samples from the RBF GP functions.
We tested them with samples from various kernel functions (Matern, Periodic inclduing
RBF) to identify that models is capable of adaptatation to new tasks.
To consider noisy environments, we artiﬁcially generate noises into the training dataset.
Our model maintains outstanding performance compared to other baselines.
In particular, when the test data comes from periodic kernel GP, which is the most different from the RBF GP, the proposed model utilizes context datasets appropriately regardless of the test data sources, whereas other models do not properly use the context data.
When comparing the likelihood values of context datasets, the proposed method preserves equivalent scores in all cases, but the others do not.
As shown in Figure 1a, the proposed method captures context points better than other methods, as only it method correctly captures the relationship even under noisy situations.
These results support our claim, the proposed method utilizes context datasets in prediction.
We report graphical explanation and the additional experimental result of 1D regression without noises in Appendix E.
4. P REDATOR -P REY M ODEL AND I MAGE C OMPLETION
We apply NPs to a domain shift problem called the predator-prey model, proposed in the ConvCNP(Gordon et al., 2019).
In this experiment, we train models with the simulated data and applied
[Table]
models to predict the real-world data set, Hudson’s bay
the hare-lynx.
Note that the hare-lynx dataset is very noisy, unlike the simulated data.
The detailed explanation of datasets is written in Appendix F.
Recent regularization methods such as Bayesian aggregation, Information dropout, and Bootstrapping are effective to improve performance.
In particular, Bootstrapping method signiﬁcantly inﬂuences predictive performance compared of other ANPs as mentioned in the original paper(Lee et al., 2020).
However, we observe that the proposed method is superior to the other models in both simulated test set and real-world dataset.
In particular, all baselines drastically decrease the likelihood values due to noises during the test, while the proposed model preserves the performance because it is robust to noises.
See the left side of Table 2.
We report the additional experimental result with periodic noises and the graphical explanation is presented in Appendix F.
Second, we conduct the image completion task in which the models generate images under some given pixels.
The experiment setting follows the previous experiments (Garnelo et al., 2018a;b; Kim et al., 2019; Lee et al., 2020).
To conduct fair comparison, all models employ only MLP encoders and decoders and the multi-head cross attention is used for local representations; the variant of ANPs.
We indicate that the proposed method records the best score as seen in the right side of Table 2.
The best baseline records 3.53 with 500 context pixels in Table 2, whereas our method attains 2.653 of likelihood even with 50 context pixels and grows to 3.948 with 500 context pixels.
Referred to Lee et al.
(2020)’s paper, the previous highest score is 4.150 of likelihood for context pixels and 3.
of likelihood for target pixels.
As Kim et al.
(2019) mentioned that the more complex architecture like the self-attention mechanism enhances the completion quality than the MLP encoders, these scores were obtained by ANP with self-attention encoders and employing bootstrapping.
However, we achieves to obtain comparable results for context points and exceed the previous highest score for target points by a signiﬁcant margins without exhausted computations and complicated architectures.
The completed images by ours and baselines are shown in Appendix J.
4. M OVIELENS -100 K DATA
We demonstrate the robustness and effectiveness of our method using a real-world dataset, the MovieLenz dataset, which is commonly used in recommendation systems.
This setting has an expensive data collection process that restricts the number of users.
In this section, we report how well the proposed method can be generalized to novel tasks using limited tasks during the meta-training.
To train NPs, we decide to split this dataset according to user ID and regard the rating samples made by one user as a task.
The purpose of this experiment is that NP provides an identiﬁer to serve for new visitor using very few rating samples.
We follow the setting used in the existing work(Galashov et al., 2019).
The proposed model performs better than the other methods.
As mentioned in subsection 4.2, it captures the information from the context dataset, while other methods suffer from noise in the data and lack of users in meta-training.
The experiment result of comaprison with baselines is reported in Appendix I.
To validate use of real applications, we compare with existing studies.
[Table] [Table]
According to Movielens-100k benchmarks, the state-of-the-art RMSE score is about 0.890, which can be obtained using graph structures about users and ﬁlms(Rashed et al., 2019).
For fair comparison, we train and test the proposed method on the same setting, named as U1 splits.
When evaluate baselines on the u1.test dataset, we randomly draw samples of corresponding users from the u1.base used in the meta-training and regard as context data points.
Although we do not use graph structures, as seen in Table 3,
The ANP points the comparable result of 0.909, meanwhile the proposed model attains a promising result, 0.895 of the RMSE value.
This experiment indicates that the proposed method can reliably adapt to new tasks even if it provides small histories, and we identify again that our model can properly work on noisy situations.
R ELATED W ORKS
The stochastic attention mechanism enables the capturing of complicated dependencies and regularizing weights based on the user’s prior knowledge.
However, such methods cannot utilize back-propagation because they do not consider the reparameterization trick to draw samples (Shankar & Sarawagi, 2018; Lawson et al., 2018; Bahuleyan et al., 2018; Deng et al., 2018).
Even if these methods employ a normal distribution as a posterior distribution of latent variables, satisfying the simplex constraints, which sum to one (Bahuleyan et al., 2018), is impossible.
Recently, the Bayesian attention module suggests that the attention weights are samples from the Weibull distribution whose parameters can be reparameterized, so this method can be stable for training and maintaining good scalability (Fan et al., 2020).
Since the Conditional neural process have been proposed(Garnelo et al., 2018a;b), several studies have been conducted to improve the neural processes in various aspects.
The Attentive neural process modify the set encoder as an cross-attention mechanism to increase the performance of the predictability and interpretability (Kim et al., 2019).
Some studies investigate NPs for sequential data (Yoon et al., 2020; Singh et al., 2019).
Trials have been combined with optimization-based meta-learning to increase the performance of NPs for the image classiﬁcation task with a pre-trained image encoder (Rusu et al., 2018;
Xu et al., 2020).
The convolutional network can be used for set encoding to obtain translation-invariant predictions specialized in image-based problems (Gordon et al., 2019;
Foong et al., 2020).
Similar to this study, the Bayesian context aggregation suggests the importance of context information over tasks(Volpp et al., 2021) and the Bootstrapping NPs orthogonally improves the predictive performance under the model-data mismatch.
(Lee et al., 2020).
Unfortunately, there is no clear explanation of how stochasticity has helped improve performance and does not show a fundamental approach to enhancing dependency between the context and the target dataset in terms of set encoding.
C ONCLUSION
In this work, we propose an algorithm, neural processes with stochastic attention.
We show that the proposed method can be explained from information theory.
We devised the new regularization to lead representations paying attention to the context dataset and help our model pay more attention to the context dataset.
A C LOSED F ORM S OLUTION FOR T HE KL D IVERGENCE B ETWEEN W EIBULL AND G AMMA D ISTRIBUTION
The generalized gamma distribution contains both Weibull and gamma distribution as special cases.
We are concerned with the three-parameter version of generalized gamma distribution introduced in Stacy (Stacy et al., 1962).
Its parameters can be categorized into one scale parameter a and two shape parameters d and p .
Its probability density function is deﬁned for x ∈ [0 , ∞ ) and given by
[Equation]
where Γ( · ) is the gamma function, and parameters a, d, p > 0 .
For d = p , it corresponds to the
Weibull distribution, and if p = 1 , it becomes the gamma distribution.
Bauckhage (Bauckhage, 2014) derives a closed form solution for the kullback-leibler divergence between two generalized gamma distribution as follow.
[Equation]
where f 1 , f 2 are the generalized gamma distributions, and ψ ( · ) is digamma function.
The Weiull distribution with scale parameter λ and shape parameter K coincides with the generalized gamma distribution a = λ and d, p = K .
For gamma distribution with scale parameter β and shape parameter α , it becomes the generalized gamma distribution a = 1 / β , d
= α and p = 1 .
The KL divergence between the Weibull distribution ( λ, k ) and the gamma distribution ( α, β ) amounts to
[Equation]
We introduce ψ (1) = − γ ≈ 0 .
5772 as Euler constant.
Finally, we obtain Equation
[Equation]
B ELBO D ERIVATIONS
In this section
[Equation]
we derive the objective function in the manuscript of this paper.
Without loss of generality, target dataset { X, Y } =
{ x i , y i } Ni =1 and context dataset D =
( X c , Y c )
= { x i , y i } Mj =
such that N (cid:29) M .
Let the log-likelihood of target data points in a task
T k be log p
( Y | X, D ) .
We begin to maximizing the log-likelihood of target data points based on context dataset and representations.
We suppose that it follows the graphical model in Figure D.2c
[Equation]
Applying Jensen’s inequality to
[Equation]
[Equation]
From Z i = { z, w i } in Figure D.2c, we deﬁne z is dependent on context dataset D and w i is dependent on target x
i and context dataset D
[Equation]
We assume that p ( w i | x i , D ) ≈
q φ 12 ( w i | K ( X c )) by regularization of latent variables to pay attention on context dataset.
It also follows the strategy of the standard neural processes : posterior distribution of z is q φ 2 ( w i |{ X t , Y t } ) and prior distribution of z is q φ 2 ( w i |{ X c , Y c } ) .
Thereby, we show that
ELBO become the objective function of the proposed method in Equation 6.
[Equation]
C P ROOF OF T HEOREM
We present that I ( y i ; D| x i ) has relation to the objective function of the proposed method.
Suppose that Z is information bottleneck for D and all variables follow the graphical models Figure 2.
As described in Equation 7, the mutual information
I ( y i , D| x i ) is to measure dependency of y i and context dataset D .
We should maximize this information to generalize the novel task.
For simplicity, we set all latent variables as information bottleneck Z for the context dataset D .
Note that
Z = ( { w i } ni =1 , z ) is information bottleneck corresponding to context representations in NPs, where
W = { w i , . . . , w n } is an attention weights and X, Y = { ( x 1 , y 1 ) , . . .
, ( x n , y n ) } is respectively data points in the target data set.
φ is regarded as the parameter of the function to transform D to the information bottleneck Z and θ is the parameter of the identiﬁer for target data point y i .
By information bottleneck theorem, Equation C.14 holds.
[Equation]
The target data point { x i , y i } is drawn by the data generating process T k ∼ p
( T ) , so that H ( y | x i ) can be regarded as a constant value due to uncontrollable factors.
As a result, − H φ,θ ( y | x i , Z ) is written as
[Equation]
We employ (cid:80) { x i ,y i } log p θ ( y i | x i , Z φ ) , which is unbiased estimate for − H φ,θ ( y i | x i , Z ) based on
Monte Carlo sampling.
Given that the concrete information bottleneck Z is given, we expect that the
Z allows us to generalize a novel task { x i , y i } by enhancing (cid:80) { x i ,y i } log p θ ( y i | x i , Z φ ) .
However, in practice, Z induced by neural networks is more likely to memorize the training dataset including irreducible noises when the number of tasks is insufﬁcient.
The conventional methods accomplished maximizing (cid:80) { x i ,y i } log p θ ( y i | x
i , Z φ ) by simply ﬁnding the relation between y i and x i only in the training dataset.
It means that the information in representation Z becomes increasingly identical to the information in x i in the meta training dataset.
It does not satisfy the condition of information bottleneck for the context dataset D .
To avoid this issue, we introduce I ( Z, x i |D ) as a regularization.
By reducing dependencies of Z and x i , we can make the information bottleneck Z focus on D .
It means that as p ( Z | x i , D ) and P ( Z |D ) get closer, the target input x i less inﬂuences Z , but D has a high correlation with Z .
In other words, minimizing I ( Z, x i |D ) is to make Z and target x i more independent given context dataset D .
It encourages latent variables
Z = (
w i , z ) to pay more attention to context dataset D
[Equation]
where, the q ( w i | x i , D ) follows the Weibull distribution and q ( w i |D ) follows the gamma distribution.
In this study, q ( w i | x i , D ) can be modeled as the stochastic attention weights and q ( w i |D ) can be modeled as the key-based contextual prior q φ ( w i | K ) .
We can factorize all latent variables because the attention weight w i does not have a dependency on data points except for the speciﬁc data point { x i , y i } .
We denote that w i has conditional independence over { w j } i (cid:54) =
j given x i .
For the global representation; z , to follow objective function of neural processes, we assume q ( z | x i , D ) =
q φ ( z | X, Y ) and q φ ( z |D ) =
q ( z |D ) .
As mentioned in section 3, we suppose that the global representation z follows normal distribution.
Therefore, we derive the regularization term
I ( Z, x t |D ) have relation to
KL Divergence terms in our loss function.
From this fact, we recognize KL Divergence terms in our objective function helps latent variables give attention on context dataset.
To summarize the mutual information between y i and D , and the newly designed regularization, we found that
[Equation]
Based on the graphical models in Figure 2 and assumptions of representations z and w i , we identify that this equation has relation to our objective function L T k
[Equation]
Finally, we can derive Theorem 1 as below
[Equation]
From Equation C.19, the gradients of φ and θ with respect to L T k can be regarded as the direction of increasing I ( y i ; D| x i ) − I ( Z, x i |D ) , where, L T k is deﬁned as the target likelihood and KL divergence of z and { w i } ni =1 in a single task.
[Equation]
where X D is input features in the context dataset and q φ ( w i | X D ) follows key-based contextual prior as described in section 3.
D I MPLEMENT D ETAILS
We referred to most of the architectures from the paper(Kim et al., 2019) and their released source code 1 .
The information dropout and importance weighted ELBO were respectively borrowed from these papers(Kingma et al., 2015; Burda et al., 2016) 23 .
The stochastic attention can be implemented based on Bayesian attention modules(Fan et al., 2020) 4 .
We migrated and revised all codes to meet our purpose.
In this chapter, we follow the notation of this paper (Lee et al., 2020).
D. (A TTENTIVE ) N EURAL P ROCESS
MLP Encoder
We suppose that multi-layers modules(MLP) have the structure as Equation D.21, where l is the number of layers, d in is the dimension of input features, d hidden is the dimension of hidden units and d out is the dimension of outputs.
[Equation]
The variants of neural processes have two types of MLP encoders: The deterministic path is used in the conditional neural process(Garnelo et al., 2018a), and the stochastic path is employed in the neural process(Garnelo et al., 2018b), attentive neural process(Kim et al., 2019) and ours.
The deterministic path is to aggregate all hidden units by the MLP encoder.
[Equation]
Instead, the stochastic path is to aggregate all hidden units and then feed-forward a single network to generate µ φ 1 and σ φ 1 .
We obtain the stochastic path r via the reparameterization trick.
[Equation]
Attention encoder We introduce cross-attention to describe the dependency of context and target dataset.
Let MHA be a multi-head attention (Vaswani et al., 2017) computed as follows :
[Equation]
Q
[Equation]
LayerNorm
[Equation] [Equation]
Where, { d q , d k , d v } are respectively the dimensions of query, key and value components.
The
LayerNorm is the layer normalization in terms of heads.
MLP Decoder
The architecture of MLP decoder is similar to the stochastic encoder.
In case of
CNP(Garnelo et al., 2018a) and NP(Garnelo et al., 2018b), this decoder transforms all representation z and the input feature { x i } i ∈ T to target distribution, the normal distribution, { µ i , σ i } i ∈ T .
[Equation]
Meanwhile, in case of attentive neural process (Kim et al., 2019) and ours, the inputs of this decoder are the global representation z , local representation { r i } i ∈ T and the input feature { x i } i ∈ T .
[Equation]
The detailed information of each architecture is described in Table D.
[Table] [Table]
3 ×- - - (cid:88) - BootstrappingANP 3 ×
Multi-heads:3 ×- - - - Ours 3 ×
Multi-heads:3 ×- - - - (cid:88)
*:InthecaseofConvNP,weproceededwith2samplesduetolackofmemory.
[Table] [Table]
In this paper, we set the dimension of all latent variables as 128, namely d h = 128 .
The number of heads in multi-head attention is 8, which is the same as the original paper of attentive neural processes (Kim et al., 2019).
For all models and all experiments, we use the Adam optimizer(Kingma & Ba, 2015) with the learning rate 0 .
001 , and we set the number of update steps as 200000 .
For training and evalation, we used AMD Ryzen 2950X(16-cores), RAM 64GB and RTX2080Ti.
The 1D regressions has a batch size of 1 and a total of 160 batches.
The Sim2Real conﬁgures the batch size to be 10 and the total number of batches to be 16.
The MovieLenz-100k has a batch size of 1 and the number of batches to be 459.
The image completion task consists of 227 batches and its size is 4.
D. M ODEL ARCHITECTURE
We graphically show the architecture of the proposed method in Figure D.1.
This model has two types of encoder as attentive neural process(Kim et al., 2019).
The encoder parameters φ consists of φ 1 , φ 2 .
The φ 1 is responsible for the local representation r i and the φ 2 is responsible for global representation z .
The encoder of global representation is same as neural process(Garnelo et al., 2018b), however, the encoder of local representation is different from the standard cross attention(Vaswani et al., 2017).
After obtaining standard attention weight w standard , we introduce reparameterization trick for w i , which follows the Weiubll distribution(Fan et al., 2020).
The important thing is that the key conceptual prior can be made by MLP φ 13 ( { x j } j ∈ C ) .
The decoder is the same as the attentive neural process(Kim et al., 2019).
D. A LGORITHM
The proposed method requires hyper-parameters k and β for reparameterization of the Weibull distribution and KL divergence between the Weibull and the gamma distribution.
We conduct the grid searches for k to ﬁnd the best value.
We identify that the proposed method with k = 300 adequately captures the dependency and generates noises to avoid memorization for all experiments.
In the case of β , we follow the setting of the Bayesian attention module(Fan et al., 2020).
We suggest the entire procedure of our algorithm as follows:
D. C OMPARISON OF PROBABILISTIC GRAPHICAL MODELS FOR VARIANTS OF NP S
We describe NPs with probabilistic graphical models to differentiate the variants of NPs.
The Neural process employs mean aggregate function and reparameterization trick to obtain a global context representation z (Garnelo et al., 2018b).
This model follows Figure D.2a.
The Attentive neural process expedites the multi-head cross attention to obtain local representation r i with global context representation z .
We show that the graphical model for ANP is the middle of Figure D.2, all variables in the attention mechanism are regarded as the determinstic variable.
In this work, we design that all latent variables contain stochasticity and achieves by the reparameterization trick.
By Bayesian attention module, we present our graphical model as shown in
Figure
Input: Task distribution p ( T ) , the shape parameters of weibull distribution and gamma distribution : k
= 300 (Image completion task : k = 100 ), β = 1 , Stepsize : γ output: encoder parameters : φ = { φ 1 , φ 2 } , decoder parameters : θ
Initialization : φ and θ randomly; while not converged do
Sample tasks {T k } Kk =1 from p ( T ) for all T k (cid:51) {T } do
Sample context dataset ( X c , Y c ) = { x j , y j } Mj =1 and target dataset ( X t , Y t ) =
{ x i , y i } Ni =1 from the task T k
Sample random noises (cid:15) 1 ∼ Unif (0 , 1) and (cid:15) 2 ∼ N (0 , 1)
Local representation { r i } Ni =1 :
Q, K, V ← f φ 1 ( X t , ( X c , Y c ))
w standard ← Softmax ( QK T / √ d k ) (Reprameterization sampling of weibull distribution for W ) λ ← w standard ∗ Γ(1 + 1 / k )
W = { w 1 , · · · , w n } ← λ ( − log(1 − (cid:15) 1 )) 1 / k
W = { w 1 , · · · , w n } ← { w i (cid:80) W i , · · · , w n (cid:80) W i } (Parameters of the prior distribution for W )
α ← f φ 1 ( K ) { r i } Ni =1 ← { w i V, · · · , w n V }
Global representation z : µ φ 1 , σ φ 1 ← Aggregator ( f φ 2 ( { x j , y j } Mj =1 ))
(Reprameterization sampling of normal distribution for z ) z = µ φ 2 + (cid:15) 2 ∗ µ φ
Decode p ( Y t | X t , z, { r i } Ni =1 ) : µ y , σ y ← ( f θ
[Equation]
X t , z, { r
[Equation]
Ni =
[Equation]
Evaluation loss :
[Equation]
end
[Equation]
end
[Figure]
E 1D REGRESSIONS
For the synthetic 1D regression experiment, we set d x = 1 , d y = 1 .
The number of layers in encoder and decoder in all baselines is respectively l enc = { l pre = 3 , l post = 1 } = 4 and l dec = 3 .
We set the dimension of latent variable as d h = 128 .
When it comes to data generation process, the training data is generated from the Gaussian process with RBF kernel.
For each task, we randomly generate x ∼ Unif ( − 2 , 2) and then y is the function value by the Gaussian process with RBF kernel, k
( x, x (cid:48) ) = s 2 · exp( − (cid:107) x − x (cid:48) (cid:107) 2 / 2 l 2 ) .
To validate our model, we establish two types of datasets in the this 1D regression experiment.
First, we set the parameters of the RBF kernel as s = 3 and l = 3 .
Hence, as shown in Figure E.3a, all datasets are drawn from continuous and smooth functions.
Second, we consider the noisy situations in the ﬁrst scheme.
We intend that the suggested scheme represents the actual real-world situation.
Unlike the existing studies (Garnelo et al., 2018a; Kim et al., 2019; Lee et al., 2020), we modify the RBF kernel function adding a high frequency periodic kernel function k ( y, y (cid:48) )
= s 2 · exp( − 2 sin 2 ( π (cid:107) x − x (cid:48) (cid:107) 2 /p )
l 2 ) .
Lee et al.
(2020) proposed a noisy situation by using random noises sampled by t-distribution; however, these noises often have exaggerated values so that the generated function does not have any tendency and seems to be entire noises.
On the other hand, the function generated by Gaussian processes with a high frequent periodic kernel is smooth but is satisﬁed with a random function every trial.
Thus, this function does not interfere with the smoothness of the RBF GP function and maintains the smoothness of all support ranges.
Therefore, we decide to use the dataset generated by the RBF GP function with periodic noises to synthetically test all baselines and the proposed method for the robustness of noises.
The sampled dataset is graphically presented in Figure E.3b.
[Figure]
To generate function values of the Gaussian process, the GPy library 5 provides various functions compatible for PyTorch 6 .
In the GPy, (cid:107) x − x (cid:48) (cid:107) 2 can be regarded as the pre-deﬁned vector { 1 , . .
. , freq } T { 1 , . .
. , freq } .
We set the parameters of the periodic kernel as freq = 30 , p = 2 π , s = 1 .
The generated functions for training datasets are shown in Figure E.3.
To test generalization to a novel task, we introduce other functions such as the Matern32 kernel GP and the periodic kernel GP as shown in Lee et al.
In this experiment, we expect the trained models to capture context points despite different functions.
Naturally, all models perform well on the
RBF GP function because the meta-training set is generated from RBF GP functions; instead, we test all baselines on other types of functions such as the Matern32 kernel GP and the Periodic kernel GP as well as the RBF kernel GP.
For the Matern32 kernel
[Equation] [Equation]
) and the periodic kernel is same as mentioned earlier.
The shared parameters of these three GPs are same as s = 3 , l = 3 .
The periodic kernel requires another parameter freq = 10 , which is the default value in the GPy.
To elaborate the meta-training framework, the minimum number of context points is 3 , and the minimum number of target points is 6 , including the context data points.
The maximum number of context points and target points is respectively 97 and 100 .
In the following subsection, we report experimental results, including clean and noisy situations.
[Table]
E. E XPERIMENT RESULT ON RBF GP FUNCTIONS
We train all baselines on the RBF GP functions without noises.
We demonstrate experimental results in terms of predictability and context set encoding.
As shown in Table E.4, all models are capable of ﬁtting the RBF GP function; meanwhile, all models are degraded in cases of Matern kernel GP and Periodic GP.
Unlike all baseline models, of which performances drop substantially, the proposed method has relatively small degradation.
Particularly, the proposed method records that the likelihood value of context points at Matern kernel GP and
Periodic GP is respectively 1.170 and 0.680.
Meanwhile, the best performance among baselines is 1.077 and 0.379 by ANP with weight decay.
This result shows that the proposed method performs better than all baselines.
The graphical results are described in Figure E.4.
E. E XPERIMENT RESULT ON RBF GP FUNCTIONS WITH NOISES
As mentioned early in the current section, we train all models on the RBF GP functions adding the random periodic noise.
Looking at the column of RBF kernel GP in Table 1, most of the baselines are degraded due to noisy situations.
However, the performances of all baselines are improved in Matern
GP and Periodic GP.
We guess that this phenomenon can be explained as mitigating memorization issues by injecting random noise to the target value (Rajendran et al., 2020).
We present how effectively random noises improve generalization performances for all baselines and our method in this experiment.
However, we can recognize that injecting random noises cannot be a fundamental solution.
As seen in Appendix G, we emphasize that capturing and appropriately utilizing context information when predicting target values is a fundamental solution to avoid memorization in a given situation.
The experimental result is described in Table 1 on manuscript and the detailed graphical explanation is suggested in Figure E.5.
All models improve performance in Matern and Periodic GP than Figure E
The outperforms baselines and regularization methods.
Even this model achieves better performance than the result on RBF GP without noises.
F P REDATOR P REY M ODEL
We follow the experimental detail in this paper(Lee et al., 2020; Gordon et al., 2019).
This experiment is designed to evaluate all baselines capable of adapting new tasks, which have slightly different task distributions.
It is called "Sim2Real".
We assume that, in this situation, we easily obtain simulation data, but it requires a high expense to collect real-world data points.
We try to train all baselines on the dataset generated by simulations and test them on real data sets that are relatively small compared to the simulation data.
The way of simulated data is generated from the Lotka-Volterra model(LV model)(Wilkinson, 2018).
Note that X is the number of predators, and Y is the number of prey at any time step in our simulation.
According to the explanation, one of the following four events should occur.
The following events require the parameter θ 1 = 0 .
01 , θ 2 = 0 .
5 , θ 3 = 1 , θ 4 = 0 .
A single predator is born according to rate θ 1 · X · Y , increasing X by one. 2.
A single predator dies according to rate θ 2 · X , decreasing X by one. 3.
A single prey is born according to rate θ 3 · Y , decreasing Y by one. 4.
A single prey dies(or is eaten)
according to rate θ 4 · X · Y , decreasing Y by one.
The initial X and Y are randomly chosen.
On the other hand, Hudson’s bay lynx-hare dataset follows a similar tendency to the LV model; however, it is oscillating time series because it has been collected in real-world records.
There were unexpected outliers, unexplained events, and noise.
For the detailed explanation, refer Gordon et al.
(2019)’s work.
All simulation codes are available on this URL 7 .
The generated simulation data can be graphically shown as the left side of Figure F.6.
In this experiment, we set d x = 1 , d y = 2 , and all remaining settings are the same as the 1D regression experiment except that the number of context points is at least 15 and the extra target points is also at least 15.
When training models, we use the dataset generated by the LV model and test on the lynx-hare dataset.
The experimental result is shown in Table
[Table]
and the prediction performance can be graphically shown in Figure F.6.
We conduct additional experiments to validate whether the random periodic noise positively inﬂuences the performance.
As conducted in the 1D regression experiment
[Table]
we utilize the periodic kernel GP function, which has very high frequency freq = 100 to add noises to the training dataset.
As shown in
Table F.
compared to Table 2, We recognize that adding random periodic noise empirically improves all baselines and our method.
Among models, our models perform best due to the full representation of context and target datasets.
The prediction results can be graphically shown in Figure F.6b
[Table]
[Figure]
G A NLAYSIS OF CONTEXT EMBEDDINGS VIA ATTENTION WEIGHTS IN 1D REGRESSIONS AND THE LYNX - HARE DATASET
The attention mechanism can explicitly display the distance between context and target data points via attention weights.
Especially, when the dimension of features is one, x ∈ R 1 , the distance between features is simply calculated and sorted, so allowing for straightforward comprehension when the attention weights are shown in heat-maps.
As a result, we present several heatmaps to compare ours with baselines in both clean and noisy 1D regression datasets.
The horizontal axis in these graphs represents the value of features in the context dataset, while the vertical axis represents the value of features in the target dataset.
The best pattern for this heat-map is diagonal because all feature values are arranged in ascending order.
Additionally, we will provide the simpliﬁed heat-map that take into account only target points with same value in the context dataset.
Due to the fact that the original version has one-hundred target points, labels and ticks are necessarily small.
We guess that readers may be unable to decipher the detailed information such as labels and attention scores.
This simpliﬁed version allows the readers to instantly grasp what we attempt to convey by exhibiting a more distinct diagonal pattern.
From Figure G.7 to Figure G.9, on the left side, we show a simpliﬁed version of the attention score; on the right side, we show the result for the entire target dataset.
Plus, we provide Sim2Real’s attention heat-maps in both clean and noisy environments.
As a result, our discovery appears to be consistent.
First, the heatmaps of the 1D regression problem without perioidc noises are shown in Figure G.7.
In this graph, all models including ours can accurately depict the similarity between the context and targete dataset.
Although the attention scores for each model vary, it is clear that the majority pattern of all models is diagonal.
Therefore, all models are trained using the clean dataset in the intended manner.
However, other phenomena are found in the noisy data.
We identify that the attentive neural process Kim et al.
(2019) fails to capture contextual embeddings because the attention weights of all target points highlight on the lowest value or the maximum value in the context dataset.
In the case of the Bootstrapping ANP Lee et al.
and ANP with information dropout, the quality of heat-map is slightly improved, but it still falls short of ours.
This indicates that the present NPs are unable to properly exploit context embeddings because the noisy situations impair the learning of the context embeddings during meta-training.
On the other hands, even in the noisy situation, the attention score in ours still appear clearly.
Second, we include heatmaps for the Sim2Real problem.
As seen in
Figure F.6, the context and target datapoints are much too many for the label information and ticks to be recognized.
Hence, we recommend that readers verify the presence of diagonal patterns rather than examining detailed numerical values.
As seen in Figure G.9a, all models are capable of capturing properly similarity between the context and target datasets.
However, as with 1D regression, the diagonal pattern of the baselines is disrupted as seen in the left graph of Figure G.9b; nevertheless, ours retains the ideal pattern in both clean and noisy situations.
When comparing our model’s heat-map pattern in clean and noisy environments, there is a noteworthy point.
Our model is capable of learning adaptively how to focus on certain context datapoints depending on the extent of dirty data in the meta-train dataset.
The model is trained on the clean dataset to take into consideration nearby points as well as the corresponding point in the context dataset, hence, the heat-map gradually changes.
This is because the clean dataset has smooth values, y i,j , near a certain feature x i,j .
Meanwhile, in the noisy dataset, the model is trained to focus exclusively on corresponding points to the context dataset.
Hence, the heat-map in Fig E.5 (b) indicates that attention score of target datapoints that includes the context dataset has a high value, whilst the remainder points treat all context datapoints uniformly.
This phenomena occurs because there is less correlation between adjacent features x i,j and its labels
y i,j in the noisy situation during the meta-training.
[Figure]
H A BLATION S TUDY : R ELATIONSHIP BETWEEN REGULARIZATION OF PAYING MORE ATTENTION TO THE CONTEXT DATASET AND CONTEXT EMBEDDINGS
This section discusses the effect of suggested regularization, paying more attention to the context dataset on attention scores.
We demonstrated how the proposed regularization term results in an increase in learning stability and embedding quality.
Prior to conducting a thorough comparison of stochastic attention with and without regularization, we analyze learning curves between baselines, such as CNP, ConvCNP and ANPs in 1D regression under a noisy situation.
Because the proposed regularization naturally incorporates the attention mechanism, we begin by comparing the proposed model to NPs that do not employ the attention mechanism.
We choose CNP with
Bayesian Aggregation, NP with Bayesian Aggregation(Volpp et al., 2021) and ConvCNP(Gordon et al., 2019) as baselines, taking into account prediction performance as demonstraed in Table 1.
As shown in Figure H.10a, the negative loglikelihood of CNP_BA and NP_BA have converged to about − 10 . 0 and 0 .
However, the ConvCNP and ours record values around − 30 .
In the case on
ConvCNP, this method optimally computes similarity between context and augmented data points (called grid ) that include target points with a high probability using the thereotically well-deﬁned functional representation by RBF kernel function.
On the other hand, the proposed regularization approaches the ideal loglikelihood value without the assistance of extra data points.
The proposed can train how to establish similarity between context and target points and predict labels of target points during meta-training.
Second, we compare training curves between ANP, ANP( dropout )(Kim et al., 2019),
Bootstrapping ANP(Lee et al., 2020) and ours.
This comparison indicates that while all methods make use of the attention mechanism, the chosen baselines exclude stochastic attention and regularization; I ( Z, x i | D ) .
When we look at Figure H.10b, we observe the same results as shown the quality of context embeddings in Appendix G. While the Bootstrapping ANP had superior training curves to ANP and ANP( dropout ), but it was unable to be match ours.
[Figure]
We examine the general effect of the proposed regularization to the stochastic attention by altering hyper-parameter K .
We conduct experiments in which all models use same generative process for stochastic attentions, as desribed in Figure 2, but with varied hyper parameter, K ∈ { 1 , 10 , 20 , 30 , 40 , 50 , 60 , 70 , 80 , 100 , 300 , 500 } and the presence of KL [ q φ ( w i | x i , X c ) | q φ ( w i | X c )] .
To begin, when we analyze Table H.6, we observe that variation of the hyper-parameter K has little inﬂuence on the prediction performance of 1D regressions.
If the models properly converge, the prediction outcomes should be statistically indistinguishable.
However, the proposed regularization has an effect on model’s learnability.
While all experiments cases that employ the proposed regularization converge, a few models with the hyper-parameter K ∈ { 10 , 50 , 70 } that do not use the proposed regularization, do not.
Additionally, we demonstrate this fact via negative log likelihood values in the meta-training, as seen in Figure H.11.
As Table H.6 indicates, all models that use the proposed regularization have converged, whereas some models that do not use the proposed regularization and use the hyper-parameter K ∈ { 10 , 50 , 70 } , have signiﬁcantly different training curves when compared of converged models.
Therefore, we anticipate that the proposed method will ensure learnability and convergence regardless of a hyper-parameter value.
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
We study the learned attention scores in detail to verify the proposed regularization’s efﬁciency.
To conduct fair comparison, we ﬁx the number of context points and choose target points that have same value as the context dataset.
The ideal scenario is that all scores in the diagonal components have identical values and little variation regardless of the placement of target points as the attention mechanism should focus exclusively on context and target dataset.
As seen in the case of no regularization in Figure H.12, despite somewhat varied scores, all converged models exhibit the same diagonal pattern.
As a result, these models present similar prediction performance.
We argue that 1D regressions do not need precise context embeddings.
However, when we compare the attention scores with and without regularization, we discover ﬁndings regarding the quality of context embeddings.
In the absence of regularization, the values of attention weights changes according to the hyper-parameter K .
Certain models place a less emphasis on the diagonal components than others, while others excessively place a greater emphasis on them.
Without the regularization, the attention scores of diagonal components range from 0.3 to 0.64.
We instantly notice that without regularization, the attention heat-map of stochastic attention has an inconsistent degree of brightness in relation to the hyper parameter K .
In particular, the heatmaps of K = 60 ,is too bright in comparison to other heat-maps, whilst the heatmaps of K = 40 is too dark.
While the diagoanl pattern is presented, there is no consistency in the brightness of any heat-maps.
On the other hand, the attention scores of regularized models exhibit little ﬂuctuation.
Regardless of the hyper parameter K , these scores are closed to 0.5.
Table H.7 contains the quantitative result.
In the regularized model, the variance of attention scores for diagonal components is 0.
and the mean value of attention scores is 0.469.
Nonetheless, the mean of the model without the regularization is 0.367 and the variance is 0.026.
This demonstrate that regularized models consistently
[Table]
properly emphasize on the context dataset according to property of context and target dataset.
The regularization cases in Figure H.12 illustrate this fact graphically.
All heatmaps of stochastic attention that employ the proposed method regularization have very a comparable brightness.
We identify that the proposed regularization ensures learning stability by ensuring the quality of context embeddings remains consistent indepedent of the hyper-parameter K .
As a consequence, we declare that the proposed method increases model’s insensitivity to hyper-parameters K by preserving the context embeddings consistently.
The intriguing fact is that the difference in attention scores for the diagonal component is rather signiﬁcant within the same models.
Since the values of target points are same as the context dataset, the attention scores should be constant.
For instance, when K = 40 and no regularization is used, the attention scores are as { 0 . 64 , 0 . 64 , 0 . 54 , 0 .
The attention scores for the same model with regularization are { 0 . 51 , 0 . 52 , 0 . 44 , 0 .
The variance in absence of regularization is 0 .
003 , whereas the variance in the presence of regulariza-
[Figure] [Figure]
tion is 0 .
As a result of the consistent scores across all target points ranges, we realize that the regularized model improves context embeddings.
The detailed analysis is shown in Figure H.14.
While the mean values of attention scores vary between individuals, the proposed regularization assures that attention scores are constantly near 0.
value.
From the perspective of NP’s aim, the proposed regularization enbales models to achieve those objectives.
The attention scores of all target points must not be inﬂuenced by feature magnitudes and their regions.
Consequently, this method has to concentrate only on the similarity between features.
The regularization aims to derive an appropriate attention mechanism that evenly weights across important regions and eliminates biases during the meta-training.
It leads to achieving task adaptation , as it equally prioritizes context data points regardless of features magnitude.
Whenever the novel arises, the regularization ensures that the models retain the quality of their context embeddings.
We investigate more research into the cases of not converged models.
As seen in Figure H.15, there are completely distinct phenomena from what we have observed so far.
When models are not converged, all attention scores remains constently uniform across all region.
As a result, the prediction performance degrades.
However, when the models use the proposed regularization, the attention scores are comparable to the others.
This means that the regularization is crucial for preventing models from trapped in the local optima.
Additionally, as seen in Figure H.11, this regularization enables models can approach to maximize log likelihood values.
By examining these phenomenon, it is clear that the approach proposed in this study performed as a regularization role.
I M OVIELENZ -100 K DATA
We follow the experimental detail in the paper (Galashov et al., 2019).
The way to split into train, test, validation dataset is described in this work 8 .
Brieﬂy, all ratings are ﬁrst divided into two datasets, which respectively have user IDs and are not duplicated each other.
We deﬁne that one is the meta-training dataset and the other is the meta-test dataset.
In this experiment, we evaluate baselines that generalize to a new customer with small history data points.
To explain a training regime, the number of context is at least three and the objective function is computed with the remaining data points.
All users do not have the same amount of ratings so that the number of data points is varying during meta-training.
When testing baselines, we sample a few ratings from unobserved users and then evaluate the likelihood values for the remaining points.
To exclude users’ unique information, we drop the User ID from the set of features and only use general features such as age, occupation.
Lastly
[Table]
in this experiment
[Table]
we set d x = 44 , d y =
[Table]
and all remaining settings are the same as the 1D regression experiment.
The detailed explanation is in Galashov et al.
(2019)’s work.
The likelihood values on the MovieLens-10k is reported in Table I.8.
[Table]
To compare RMSE scores with existing methods for Movielens-100k data, we suggest evaluating
NPs using u1.base and u1.test.
As mentioned, the prediction by NPs is based on given context data points.
First, we randomly choose a customer in the u1.test.
Second, we randomly sample a small number of ratings from u1.base, which is used as the meta-training data.
In this experiment, we decided to draw ten samples from u1.base and employ these points as the context data set.
The ten ratings is relatively a short history compared to the size of the u1.base.
Third, we perform predictions for the data points in the u1.test based on the context data points.
The important thing is that context and target data points are sampled from the same user but the different data set.
Plus, we do not touch in information in the test set, at least.
Based on the relationship between sampled context points and target points, the performance of NPs is somewhat variable, so we report ﬁve runs and average these results.
The experimental result is in Table 3.
J I MAGE C OMPLETION
In the image completion task, we follow the training regime suggested by NPs(Garnelo et al., 2018a;b).
In this regime, d x means the pixel location and d y means the color RGB data.
Namely, we set d x = 2 and d y = 3 .
For high capacity representation, the dimension of all hidden units is determined as d h = 512 .
All remaining settings, including layer sizes, are the same as the 1D regression experiments.
When it comes to image completion tasks, according to these papers, NPs are known to complete facial images given partial pixels.
Although a few pixels are provided, NPs are capable of understanding the common appearance of a human face.
These models generate eye, mouth, and nose even if only a small number of local information is given, which is generally hard to recognize.
The Conditional neural process properly performs; however, it often loses its context representations.
The Attentive neural process tackles this issue to employ an attention mechanism.
Instead of the encoders consisting of MLP layers, the self-attention can improve the quality of pixels generated at the context data points.
However, as mentioned on subsection 4.2, scores of generated pixels in unseen regions are still problems.
Even though many pixels are provided, the quality of completed images is not improved.
In this work, the proposed method concentrates on given contextual pixels and complete images based on that information.
Hence, the proposed method completes an image with high uncertainty when few pixels are given.
Plus, it can be seen that the higher the pixels information, the higher the quality of generated images.
The numerical result is presented in Table 2, the graphical evidences are shown in from Figure J.16 to Figure J.19.
The proposed method allows for completion of images using only partially provided pixels, so it enables to complete larget images than the image size in the training dataset.
We produce images of various sizes using the proposed method, which is trained on the CelebA dataset with size of 32 × 32 .
The completed images are presented in Figure J.20. When we qualitatively evaluate the completion results, the proposed method properly perform for generating large images without sacriﬁcing performance. All completed images are quite close to the ground truth. As a result, we conclude that the proposed method can be used for task adaptation since it can complete images that the model have never encountered during meta-training and whose sizes differ from the training dataset.
In particular, while the percentage of contributed pixels to the total number of pixels remains constant, the quality of completed images are enhanced according to the amount of provided pixels.
We understand that the number of pixels is more signiﬁcant than the ratio of context points.
This is because the model has a sufﬁcient pixel counts to work with when images are large.
For instance, employing only 300 pixels as a context dataset is insufﬁcient to complete the images.
Nonetheless, the completed images with 300 pixels are comparable to those with higher pixel counts and is of higher quality than the baselines in Figure J.18.
There is an interesting discovery of uncertainty values in the image completion task.
Basically, we expect that many pixels reduce the uncertainty of generated images.
However, the existing methods stably generate facial images, but uncertainty values do not decrease even if available pixels increase.
The proposed method makes high variance for a completed image under the small number of context points.
As shown in Figure J.21, it can decrease the uncertainty of completed images when the number of context points is large.
From this fact, we identify that the proposed method performs in an intended manner.
[Figure]