A TTENTIVE S EQUENTIAL N EURAL P ROCESSES
Anonymous authors
Paper under double-blind review
A BSTRACT
Sequential Neural Processes (SNP) is a new class of models that temporally extend Neural Processes (NP) and can meta-learn a sequence of stochastic processes.
This learned function however under-ﬁts the provided contexts as is also the case in NP.
Applying attention to contexts serves as a solution but doing the same in SNP is challenging because it cannot directly access and attend over the past contexts.
In this paper, we propose Attentive Sequential Neural Processes (ASNP) which resolves the under-ﬁtting in SNP by introducing a novel imaginary context, modeled as a latent variable, over which the attention can then be applied.
We evaluate our model on 1D Gaussian Processes regression and 2D moving MNIST/CelebA regression.
We apply ASNP to implement Attentive TemporalGQN and further evaluate on the moving CelebA task.
I NTRODUCTION
Neural Processes (NP) combine the strengths of neural networks and Gaussian processes (GP) such that, like Gaussian processes, it can ﬂexibly learn a new stochastic process while at test time still providing fast O (1) prediction speed like neural networks.
Learning from small datasets from different tasks (i.e., different stochastic processes), NP can also be seen as a probabilistic latent variable framework for meta-learning.
Sequential Neural Processes (SNP) (Singh et al., 2019) are a class of sequential latent generative models that extend the power of neural processes (NP) to a sequence of stochastic processes.
SNP targets problems where the sequence of stochastic processes are governed by an underlying transition dynamics and thus learning to transfer this temporal trend between stochastic processes are useful.
Combining the meta-learning aspect of NP with temporal transfer,
SNP can be seen as a meta-transfer learning model (i.e., transfer learning of meta learning) or (temporal) stochastic process of stochastic processes.
A well-known problem of NP is underﬁtting: summarizing all context data into a single latent vector by a permutation-invariant encoding, it is quite easy to lose the details of the individual datapoint in the context.
To resolve this problem, Kim et al. (2019a) proposed Attentive Neural Processes (ANP).
In ANP, a query-dependent representation is obtained by applying an attention mechanism to the context data points with a query data as the key.
Therefore, when making a prediction on a query, the model utilizes not only the global summary of the NP latent but also is informed by the query-sensitive representation in which the details of individual data points are provided through attention, mitigating the underﬁtting problem.
Providing a single global latent for all the past and current context, SNP is also subject to the underﬁtting problem, perhaps more severely.
One may consider applying attention to SNP in a similar way as ANP does.
However, as a sequential model which, unlike ANP, does not store any of the past context but compresses them into a single encoding through an RNN, it is quite unclear how to realize an attention mechanism in SNP.
Although we can still apply attention limitedly to the context of the current time step, in SNP we assume this context is a very small amount and even empty.
Therefore, at ﬁrst glance, augmenting SNP with an attention mechanism may seem somewhat an ill-posed problem.
In this paper, we propose to resolve this problem by introducing imaginary context that augments the context set, as latent variables, and then apply attention on the union of the real context and inferred latent context.
We call the proposed model as Attentive Sequential Neural Processes (ASNP).
ASNP consists of four modules: (i) context imagination, (ii) global context-encoding without attention, (iii) local attentive context-encoding, and (iv) prediction.
ASNP uses attention at two-levels.
It ﬁrst obtain the imaginary observations using attention and then make the prediction using attention over the imaginary context given a query.
In experiments, we demonstrate the effectiveness of applying this attention mechanism into SNP for several scenarios of a variety of tasks: 1D Gaussian Process
Regression, 2D moving MNIST/CelebA
Regression and 2D moving CelebA rendering.
The contributions of the paper are as follows.
We ﬁrst introduce a new model, Attentive Sequential
Neural Processes, that resolves the problem of augmenting attention mechanism on SNP.
Then, we demonstrate that the proposed model can resolve the underﬁtting problem in SNP in various scenarios and tasks.
B ACKGROUND
Neural Processes (NP) (Garnelo et al., 2018b) learns the process of learning a stochastic process, mapping an input x ∈ R d x to a random variable y ∈ R d y , using a set of context samples C
=
( X C , Y C )
= { x i , y i } i ∈I ( C ) .
Here, I ( C ) is the set of indices for the elements in set C .
Speciﬁcally, to learn a stochastic process from a context, NP uses the conditional prior P ( z | C ) and then it can use the representation z ∼ P ( z | C ) of the inferred stochastic process to make prediction y on query x .
This is modeled by p ( y | x, z ) .
The full generative process of NP can be written as:
[Equation]
[Equation]
) where P ( Y | X, z ) = (cid:81) i ∈I ( D ) P ( y i | x i , z ) and D is the target observations D =
( X, Y )
= { x i , y i } i ∈I ( D ) .
The training data for NP is obtained by iterating sampling a stochastic process and then sampling ( C, D ) from the sampled stochastic process.
Thus, considering a stochastic process as a task, NP can be seen a probabilistic meta-learning framework.
The Generative Query Networks (GQN) (Eslami et al., 2018) are an application of NP to the problem of learning representation and rendering of 3D scenes from a set of partial 2D observations.
In GQN, a query x becomes a camera viewpoint in a 3D environment and an output y corresponds to an image taken from the viewpoint x .
The scene representation of the original GQN is querydependent and thus can make inconsistent generations across queries and thus in the following we use Consistent GQN (CGQN)
(Kumar et al., 2018) that resolves this problem by making the scene representation query-independent.
For simplicity, in the following our use of the term GQN actually refers CGQN.
The Attentive Neural Processes (ANP) are proposed to resolve the underﬁtting problem of NP by (Kim et al., 2019b).
To this end, ANP has two context-encoding paths.
One path is the same as the scene representation of NP summarizing all context information independently to the target query x .
The other path is attentive encoding of the context which is dependent to the target query x
due to its usage as the attention query.
Because the detail information of speciﬁc contexts can be preserved with attention, this results in mitigating the underﬁtting problem of NP.
Sequential Neural Processes (SNP) proposed by (Singh et al., 2019) introduce temporal transition of underlying stochastic processes.
Thus, SNP can be seen as a stochastic process of stochastic processes or meta-transfer learning.
This transition modeling is done by introducing temporal prior
P ( z t | z <t , C t ) that depends on (i) the progress of the past stochastic processes from which we learn the general temporal trend and (ii) the context of the current time step which provides a quick metalearning ability for the current stochastic process.
Then, given the updated representation z t , we can make predictions in the same way as in NP, i.e., P ( y t | x t , z t ) .
Combining these and extending to a time-horizon, we have the following generative model of SNP:
[Equation]
Applying SNP to the problem of GQN, Singh et al.
(2019) also proposes Temporal Generative
Query Networks (TGQN).
As an extension of NP toward temporal modeling, SNP is also prone to the underﬁt the context as does NP.
Therefore, it can be seen as a natural direction to apply attention to SNP to resolve the underﬁtting problem.
A TTENTIVE S EQUENTIAL N EURAL P ROCESSES
How can we augment SNP with an attention mechanism?
This problem might look simple at ﬁrst glance, provided that there is already the ANP model available, is actually not quite straightforward.
This is because, unlike the ANP which assumes all context points are stored in a memory, SNP assumes that it cannot store the past context as it is.
This means that, in SNP the entities to apply attention on is severely limited only to the context of the current time-step.
This is problematic because we assume that the context per step is small or even empty.
In such a setting, attention is not effective in general.
To resolve this problem, we introduce the imaginary context , and thereby augment the entities on which attention can be applied.
This imaginary (or pseudo) context observations are treated as latent variables and are generated by an RNN that encodes all of the past context, both the real and the imaginary.
Thus, the role of the imaginary context is to infer “ which data points other than the real context would be helpful if we were able to refer to those for attention and prediction? ”
Similarly, it can also be seen as a dynamic and conditional restoration (decoding) process of context after summarizing (encoding) the past through an RNN.
After this restoration, we can apply attention on the extended context set.
In the following, we now describe the proposed model ASNP with particular focus on (i) how to generate the imaginary context and (ii) how to apply attention thereafter.
3. G ENERATIVE P ROCESS
[Figure]
The generative process of ASNP can be decomposed into four modules: (i) context imagination, (ii) global context-encoding, (iii) local (attentive) context-encoding, and (iv) prediction.
The purpose of context-imagination at each step t is to generate an imaginary context set
C it =
( X it , U it ) which consists of imaginary queries
X it =
{ x it,k } Kk =1 and its corresponding observations (or its representation) U it =
{ u it,k } Kk =1 .
That is, we treat these imaginary context as latent variables.
The generation is conditioned on the real context C rt and all the past contexts C <t (where C t is the combined context C rt ∪ C it ) through an RNN encoding.
That is, it uses the current real context as a query to generate an imaginary context.
The generation of imaginary observation is conditioned on the imaginary query, and attention over the real context is used.
We explain this attention in more detail in the next section.
Then, the whole context-imagination model can be written as:
[Equation]
An interesting interpretation on this modeling can be observed in comparison to the memory retrieval process in the human brain (Eichenbaum, 2017).
As it is well-known in neuroscience, “ human memory is not a literal reproduction of the past, but instead relies on constructive processes that are sometimes prone to error and distortion ” (Schacter, 2012).
Our model resembles this process in the sense that it (i) compresses the past observation and importantly what it has imagined in the past through an RNN (which can be considered as lossy memory consolidation), and then after observing current context, (ii) recalls from the compressed memory not a simple copy of the past experiences but a constructive recreation of representations that is directed to help prediction.
In global context-encoding , we obtain the global representation z t which provides the representation of underlying stochastic process independently to queries.
This module can be considered the prior in the standard SNP where we do not use attention and can be written as P ( z t | z <t , C t ) .
Note that it is a design choice whether to choose C t or C rt for the conditioning context of this module.
If we use C rt , z t becomes global context-encoding of real contexts.
Otherwise, it becomes encoding of the combined contexts.
The local context-encoding is a query-dependent encoding using attention over the combined context.
Here, we use the query x t as the attention query, and the ( x, y ) ∈ C t pairs in the combined context as the key-value slot to attend.
This is a deterministic function that gives us the attention encoding r x t = f attend ( x t , C t ) which we describe in more detail in the next section.
Lastly, the prediction model P ( y t | x t , z t , r x t ) can be implemented simply by providing additional vector r x t to the likelihood function SNP.
Combining all, we can write the full generation process of ASNP as follows:
[Equation]
3. A TTENTION WITH I MAGINATION
[Equation]
In ASNP, we use attention in two modules.
The ﬁrst attention is to obtain the imaginary output C it .
For this, we ﬁrst generate X it from an RNN that takes the previous imaginary inputs
X it − 1 and the context encoding v rt = f order-inv ( C rt ) as the input and updates its state to h Xt .
Then, X it is sampled from h Xt , implementing (cid:81)
Kk =1 P ( x it,k | C <t , C rt ) .
Given this generated imaginary input X it , the imaginary output-encoding u it,k is generated as follows.
First, for each imagination k ∈ { 1 , . .
, K } , we maintain an imagination-tracker RNN, which is denoted by h ut,k and takes ( x it,k , u it − 1 ,k ) as input.
Using the union of the imaginary context-pairs and the real context-pairs, S it = { ( x it,k , h ut,k ) } ∪ { ( x rt , f y → u ( x rt , y rt )) } , as the key-value set, we apply attention on the set S it with x it,k as the attention query, and obtain the attention encoding a it,k .
Then, we sample u it,k ∼ N ( f uµ ( a it,k ) , f uσ ( a it,k )) and complete the module P ( u it,k | x it,k , C <t , C rt ) .
The second attention is implemented in the prediction module P ( y t | x t , z t , C t ) .
Given x t as the attention query, we use C t , the union of the real C rt and imagined C it context, as the key-value set (after encoding each context x ∈ X rt and y ∈ Y rt using f y → u ).
The attention encoding r x t = f attend ( x t , C t ) is then concatenated with the global encoding z t to make an input for the prediction y t .
Algorithm 1 describes an algorithm about the imagination update and attention encoding.
3. L EARNING AND I NFERENCE
Due to the intractability of the true posterior, ASNP is trained via variational approximation with the following posterior approximation:
[Equation]
where Q ( C it | C
i<t , C r , D ) =
Q ( U it | X it , C i<t , C r , D ) Q
( X it | C i<t , C r , D ) and D =
( X, Y ) .
For training, the following evidence lower bound (ELBO) is maximized w.r.t.
θ and φ :
[Equation]
For backpropagation, we use reparameterization trick (Kingma & Welling, 2013) for continuous variables.
Derivation of eq.
6 is in Appendix A.
R ELATED W ORKS
Modeling stochastic process on few data set is attractive recently related with meta-learning.
Conditional Neural Processes (CNP) (Garnelo et al., 2018a) models a regression function without global latent, which causes inconsistency between queries.
NP (Garnelo et al., 2018b) resolves it by proposing an explicit global latent. While CNP and NP are designed for more simple task, GQN (Eslami
19 − 1 .−− 0 .− 0 .− 0 .0 .0 .
Time
T a r g e t N
LL 1DGPRegressionScenario(a)      19 − 1 .−− 0 .− [-
-]0 .− 0 .0 .0 .
Time 1DGPRegressionScenario(b) 1 6 11 16 21 26 31 36 41 46 −− 0 . 0 . 1 .
Time 1DGPRegressionScenario(c)
NPANPSNPASNP et al., 2018) develops it to render 3D scene.
Like CNP, GQN is also inconsistency between queries due to absent of a global latent and CGQN (Kumar et al., 2018) resolves it with a global latent.
In a line to enhance performance, ANP (Kim et al., 2019b) resolves the underﬁtting with attention for queries and Rosenbaum et al.
(2018) applies attention on GQN model by attention given raw images as a query to render complex 3D scene (i.e. Minecraft).
SNP (Singh et al., 2019) is proposed as a generalized model for sequential stochastic processes with TGQN for rendering the dynamic 3D scene.
Functional Neural Processes (FNP) (Louizos et al., 2019) is proposed to learn a graph of dependencies between pre-selected points and training points for modeling distributions over functions.
In aspect that imaginary context is pseudo and few number of trainable representative points, it is related to inducing points of Set Transformer (Lee et al., 2019) and sparse GPs (Snelson & Ghahramani, 2006; Titsias, 2009).
VampPrior (Tomczak & Welling, 2017) also uses few trainable pseudoinputs to resolve overﬁtting, over-regularization and high computational complexity.
The reference set of FNP to model a graph of small number of points is also motivated from inducing points of sparse GPs.
In aspect of using trainable memory, the differential neural dictionary (Pritzel et al., 2017) stores and updates the previous trajectories as (key, value) pair to learn fast a variety of environments.
E XPERIMENTS
We evaluate ASNP on the following selection of tasks: a) 1D Gaussian Process (GP) regression b) 2D moving MNIST (LeCun et al., 1998) and 2D moving CelebA
(Liu et al., 2015).
On these tasks, we evaluate and compare the proposed model against NP, ANP, SNP as the baselines.
Each task is evaluated with three scenarios, which is used in (Singh et al., 2019) for 1D GP regression: a) To evaluate the model’s ability to extrapolate the future, we provide high-amount of context in the ﬁrst 10 time-steps out of the 20.
b To test the model on its ability to track the dynamics using intermittently arriving context, we provide high-amount of context in 10 randomly chosen time-steps out of the 20.
(c) To test the model’s ability to gather and make use of low amount of context received over long segments of time, we provide low amount of context in 45 randomly chosen time-steps out of the 50.
In each setting, the remaining time-steps are used for prediction.
The details about model architectures can be found in Appendix B.1.
To test the beneﬁts of attention for rendering 2D images, we used moving CelebA because it has higher uncertainty when given partial knowledge in comparison to an alternative data set such as moving simple shapes (e.g. circle). We test on the scenario (c) as described above, but with shorter
− 1 .− 1 .−− 0 .− 0 .
Time
T a r g e t N
LL 2DMovingMNISTRegressionScenario(a)      19 − 1 .− 1 .−− 0 .− 0 .
Time 2DMovingMNISTRegressionScenario(b) 1 6 11 16 21 26 31 36 41− 1 .−− 0 .− 0 .− 0 .
Time 2DMovingMNISTRegressionScenario(c)
NPANPSNP
ASNP(k=9)ASNP(k=25)ASNP(k=100)      19 − 3 .−− 2 .−− 1 .−
Time
T a r g e t N
LL 2DMovingCelebARegressionScenario(a)
−− 2 .−− 1 .−
Time 2DMovingCelebARegressionScenario(b) 1 6 11 16 21 26 31 36 41−− 2 .−− 1 .−
Time 2DMovingCelebARegressionScenario(c)
NPANPSNPASNP sequence lengths and smaller context sizes.
More details are explained in the section on 2d rendering.
1D regression: We ﬁrst evaluated our method on sequential 1D Gaussian Process data set.
This is an extension of the GP data set through the addition of linear dynamics on the kernel hyper-parameters as used in (Singh et al., 2019).
In each transition, we add a small Gaussian noise to introduce stochasticity.
For scenarios (a) and (b), the number of context observations n is randomly selected in [5 , 50] or an empty set.
The number of target observations m is randomly selected in [1 , 51 − n ] .
For scenario (c), n is 1 or 0, and m is chosen from [1 , 11 − n ] .
The target set subsumes the context.
The more details about this task is described in Appendix C.
In Fig 2, SNP outperforms NP as reported in (Singh et al., 2019).
On the other hand, in scenario (a), the performance of ANP degrades steeply after the context are removed since it does not model the dynamics explicitly.
Even though it appears to outperform NP and SNP, this can be credited to the attention in ANP that prevents underﬁtting (Kim et al., 2019b).
ASNP shows better performance for all the scenarios, outperforming ANP with a greater gap in task (c).
We hypothesize that when dense context is given at each time-step in (a) and (b), ANP can ﬁt well as it does not need to rely on the dynamics much.
But in task (c), the context is low in each time-step and the model must capture dynamics to make best use of the context seen thus far.
In Fig 3, we illustrate a sample of validation set in task (c) with predictions from ASNP and the baselines.
As shown, even for the points not shown recently, ASNP predicts better than the baselines with lower uncertainty.
ANP and SNP fail to predict those points well.
This shows that the update of the imaginary queries and the representations is necessary over and above the sequential update of the global representations or the expensive modeling of the entire past using the time information.
More qualitative results are in Appendix E.1.
2D regression: We take a step further to evaluate our model on a more challenging 2D setting where the queries are the pixel locations and the regression is over the pixel values of images.
The image is a 42 × 42 -sized black canvas with a moving MNIST digit (sized 28 × 28 ) or a cropped CelebA face (resized to 32 × 32 ). The digits or the faces start moving from a random location heading towards a direction randomly selected at t = 0 with a speed of 3 pixels per time-step and bouncing perfectly upon encountering a wall.
Like in the regression task, a small Gaussian noise is added to the location obtained after each transition.
Same as the regression task, 3 context regimes are designed.
In regime (a) and (b), n is either 0 or n ∈ [5 , 500] while m is drawn from [1 , 501 − n ] .
In (c), n is either 0 or 30 and m is drawn from [1 , 51 − n ] .
The target set subsumes the context.
We present the quantitative results for 2D regression tasks in Fig.
From these, we infer the following points. We observe that SNP performs better than ANP except in the early time-steps of scenario (a) when a large number of contexts points are shown without interruptions.
We reason that in the presence of such ample context points, NP and SNP still suffer from under-ﬁtting while ANP can make an effective use of those points without having to model the dynamics. But when the task setting demands that the model capture the dynamics, as in task (c), both NP and ANP perform poorer than SNP and ASNP.
This is more pronounced in the 2D setting than in the 1D because tracking dynamics of images without explicit temporal modeling is harder. We further note that
MNIST digits are simpler than the CelebA faces. Therefore in the latter case, SNP struggles with underﬁtting and ANP struggles with modeling the dynamics, causing them both to furnish similar performances in task (b).
ASNP does not suffer from either of these problems and it outperforms all baselines in all the tasks.
In scenario (a), we also note a sudden performance drop in NP, ANP and ASNP.
While unsurprising in NP and ANP, it also occurs in ASNP due to sudden switch-over to a purely imaginary context. This drop does not appear in the 1D tasks since the number of context points provided are fewer and therefore ASNP is less impacted by their absence.
We also evaluate the effect of varying the imaginary context sizes on the moving MNIST data set for the scenarios (a) and (b).
We observe that a larger k yields better performance. This gain, however, saturates quickly as the improvement between k = 9 and k = 100 is not large.
This shows that the imaginary context can represent the stochastic process with only a few points.
In Fig. 5 and Fig. 6, we present the qualitative results on the two data sets for scenario (c).
Here, we note that NP and ANP can produce the digit location but not the appearance.
SNP captures the appearance of the digit but not clearly.
Although ASNP also produces fuzzy appearance in the early time-steps, the digit becomes identiﬁable by t = 10 and the details keep getting clearer over t = 20 .
We also see a similar trend for the moving CelebA task.
More illustrations of scenarios (a), (b) and (c) are included in Appendix E.2.
Time T a r g e t N LL 2D Moving CelebA scene rendering scenario (c) GQNTGQNATGQN
2D scene rendering: We also test ASNP with the application of ASNP to a GQN architecture on the moving CelebA rendering task.
We call it Attentive TGQN (ATGQN).
To design this, we use the Temporal-ConvDRAW as in TGQN (Singh et al., 2019) for encoding a global latent.
For computing a query-dependent representation, we apply the ASNP’s attention encoder with imagination to get a scene-wise matrix.
We use the same decoder as in TGQN.
While it is a limited solution working on the task where y t can be divided as points with queries, it resolves the underﬁtting for overlapping pixels between the context and the target.
Unlike NPs, partial overlap exists, which is difﬁcult to predict by scene-wise attention.
More details about the model architecture is described in Appendix B.2.
The canvas size of the data set is 80 × 80 and cropped face image size is resized in 64 × 64 .
The direction of motion and initial position are randomly selected at t = 0 with 13 pixels per time-step speed.
The size of output y is 32 × 32 .
The sequence length is 25.
One context view is provided to the model at each of the randomly chosen 20 time-steps out of 25 time-steps. and an empty set is given on the remaining 5 time-steps.
The target size m is drawn in the range [1 , 11 − n ] .
In Fig.
7, we show quantitative and qualitative results for scenario (c).
We can see that ATGQN outperforms TGQN because for the overlapping areas, ATGQN resolves the underﬁtting.
The performance gap between TGQN and ATGQN, however, is smaller compared to the previously seen 1D and 2D regression tasks.
A similar trend is observed in scenario (a) whose results are in Appendix D.2.
The reason for this is low uncertainty in the presence of the context that provides a wide view of the canvas. This and the small number of context causes lesser underﬁtting than what would have arisen had the context observations been highly partial as in the regression tasks.
More examples are included in Appendix E.3.
C ONCLUSION
In this paper, we addressed the problem of underﬁtting that plagues Neural Processes and Sequential
Neural Processes.
Although this is resolved in the former by ANP, it is not possible to resolve it in SNP with direct application of attention since SNP cannot store the past context.
We introduced Attentive Sequential Neural Processes which comprises a novel memory mechanism of imaginary context to resolve the underﬁtting.
It not only compresses the past knowledge but does it in a way that is geared towards making good predictions with low uncertainty.
In the experiments, the proposed model shows superior performance on various tasks for a number of sequential scenarios.
Since ASNP is a model based on attention, its limitations include those of any attention-based model.
One such case is the scene-wise attention.
Scene-wise attention cannot ﬁt properly for partial overlapped region between context and target.
Although we partially solve it with pixel-wise attention, it would still be interesting to encode query dependent representation for scene observation
A PPENDIX A ELBO D ERIVATIONS
[Equation]
where Q φ ( z t , C it | C r , D ) =
(cid:81) tt (cid:48)
=1 Q ( z t (cid:48)
| z <t (cid:48) , C it (cid:48) , C r , D ) Q ( C it (cid:48)
| C i<t (cid:48) ,
C r , D ) and KL expectation
Q φ ( z <t , C i<t ) = (cid:81) t − 1 t (cid:48)
=1 Q ( z t (cid:48) , C it (cid:48)
| z <t (cid:48) , C i<t (cid:48) , C r , D ) to simplicity.
[Figure]
[Figure]
Every models share a basic architecture (e.g. context encoder and decoder).
They have two encoders.
One of them is for a global latent.
It consists of 3 layers MLP with ReLU (Nair & Hinton, 2010) is used as encoder and 2 layers MLP is used to encode and sample a latent.
Another encoder is consisted of 6 layers MLP with ReLU activation function and mean function for NP and SNP and attention module for ANP and ASNP.
Decoder is consisted of 4 layers MLP with ReLU activation function and 1 layer MLP is used to sample ˆ y with uncertainty.
For non sequential models (NP and ANP), time is encoded as a normalized ﬂoat scalar, e t = 0 .
25 + 0 .
5( t/T ) , where T is the length of sequence of data.
After encoding, it is appended in query as x (cid:48) =
( x, e t ) .
For attention models (ANP and ASNP), Multihead attention (Vaswani et al., 2017) is used because it showed the best performance in a variety of attention methods (Kim et al., 2019b).
ASNP has two attention modules, which shares parameters.
For sequential models (SNP and ASNP), they uses same temporal architecture for a global latent.
LSTM with a default setting of TensorFlow (Abadi et al., 2016) is used and the dimension of hidden unit of LSTM is d r where d r is the dimension of representations.
For ASNP, two RNN modules for the imaginary queries and representations is LSTM with a default setting of Tensorﬂow.
The dimension of hidden unit of LSTMs is d r .
Here, the number of state of LSTM for representations is k × batch size.
The dimension of representation d r is 128 and the dimension of query d q is 1 for 1D regression and 2 for 2D regression.
Note that for non-sequential model, d q is 2 and 3 by attaching encoded time e t .
The number of imaginary context is 25.
The initial imaginary context C i 0 is trainable parameters.
Learning rate is 0.0001 and batch size is 16 for 1D regression and 8 for 2D regression.
The diagram for ASNP architecture is in Fig.
B. GQN S
For an encoder, tower representation (Eslami et al., 2018) is used.
A decoder is similar between models (GQN, TGQN and ATGQN) based on the decoder of CGQN (Kumar et al., 2018) that is a deterministic generative model working on auto-regressive manner.
The different is inputs that are described in bellows.
For GQN, time is encoded and appended in query same to NP and ANP.
To sample z t , we apply a convolutional DRAW(ConvDRAW)
(Gregor et al., 2015; 2016) like GQN (Kumar et al., 2018).
To render ˆ y , the decoder inputs z t , X t and v rt that is an output of the decoder.
v rt is averaged representation that is an input for every target queries.
For TGQN, Temporal-ConvDRAW (Singh et al., 2019) is applied to sample z t with ConvLSTM (Xingjian et al., 2015) state, h t .
We take RSSM to implement as (Singh et al., 2019).
The decoder input is a set of the global latent z t , the target queries X t and ConvLSTM state h t , in which, z t and h t are independent to target queries.
For ATGQN, a global latent is sampled same to TGQN and a query dependent representation of ASNP is used.
To make a pixel-wise path, we split scene view as pixel view with queries per each pixel.
When many context are given, overlapped between context exists.
We leave it to optimize batch operation.
We encode the pixel-wise representation with 2 convolutional layers to make same dimension with scene-wise, the dimension of which is [ scene-height / 4 , scene-height / 4 , 3] .
The input of decoder are z t , X t and r t = { r ti } i ∈I ( D ) that is query dependent representations for each target query x ti .
The dimension of representation d r is 128.
The number of steps for (Temporal) ConvDRAW to sample z t is 3 and the dimension of z is 4.
Note that z t is the output of the last roll-out of (Temporal) ConvDRAW.
For TGQN and ATGQN, the number of hidden unit for ConvLSTM is 40.
The dimension of hidden unit on the decoder is 32 and the number of steps for auto-regressive decoding is 6 and output is cultivated.
The query size is 2 (for GQN, it is 3 with e t ).
The number of pseudo context is
A PPENDIX C G AUSSIAN P ROCESS D ATA S ET
The kernel hyper-parameters, length-scale l and kernel-scale σ are selected randomly at t = 0 in [0 . 7 , 1 . 2] and [1 . 0 , 1 . 6] for scenarios (a) and (b).
For scenario (c), l and σ are chosen in [1 . 2 , 1 . 9] and [1 . 6 , 3 . 1] at t = 0 .
The true underlying dynamics of the kernel hyper-parameters ∆ l and ∆ σ are in [ − 0 .
03] and [ − 0 .
05 , 0 . 05] chosen at t = 0 .
Gaussian noise on the dynamics is ∼ N (0 , 0 . 1) .
A PPENDIX D A DDITIONAL E XPERIMENTAL R ESULTS D. L EARNING C URVE ON GP TASK
[Figure]
NPANP
SNPASNP
In this section, we show target NLL for iterations (Figs. 9) and wall clock time (Figs. 10) for 1D
GP data set.
ASNP and ANP computation complexity is bigger than NP O (1) due to attention.
ANP needs O ( m (cid:80)
Tt n t ) where n t is the context size at time-step t .
Because it needs to attention to entire context in current.
ASNP computation complexity is O (( n + k ) m ) for generating r t and
O (( n + k ) k ) for updating the pseudo context.
When the sequence of data T is not long, ANP computation complexity is smaller than the complexity of ASNP.
However, Fig.
10 shows ASNP quickly saturates on lower loss in same time.
5 · 10− 1 .−− 0 .− 0 .− 0 .0 .0 .
Seconds(inhundredthousands)
T a r g e t N
LL 1DGPregressiontask(a)
{+5 · 10− 1 .−− 0 .− 0 .− 0 .0 .0 . Seconds(inhundredthousands) 1DGPregressiontask(b) 0 .0 .0 .0 .
 · 10−− 0 . 0 . Seconds(inmillions) 1DGPregressiontask(c)+}
NPANPSNPASNP
D. 2D RENDERING TASK FOR SCENARIO ( A )
We also test our model on the moving CelebA rendering task for scenario (a).
The sequence length
T sets as 15 and the context is given to t = 7 .
The context size n is randomly selected in [1 , 10] or 0.
The target size m is randomly chosen in [1 , 11 − n ] .
Other environment setting is same to scenario (c).
[Figure] Fig.
11 shows results that are both quantitative and qualitative.
Target NLL value of GQN is very high.
On the other hand, the generation performance is not the same as the target NLL.
The explanation for this is inability to predict the complex face image’s location.
When context is given,
ATGQN outperforms TGQN because ATGQN solves the overlapping region underﬁtting.
ATGQN however shows similar outcomes to TGQN on non-context prediction.
The explanation for this is less uncertainty due to a lot of contextual information.
[Figure] [Figure] [Figure]