A TTENTIVE S EQUENTIAL N EURAL P ROCESSES
Anonymous authors
Paper under double-blind review
A BSTRACT
Sequential Neural Processes (SNP) is a new class of models that temporally extends Neural Processes (NP) and can meta-learn a sequence of stochastic processes.
This learned function however under-ﬁts the provided contexts as is also the case in NP.
Applying attention to the contexts resolves this in NP but simply extending this to SNP and applying attention on a buffer of context history is sub-optimal as our ﬁndings show.
In this paper, we propose Attentive Sequential
Neural Processes (ASNP) which resolves the under-ﬁtting in SNP by introducing a novel imaginary context, modeled as a latent variable, over which the attention can then be applied.
We evaluate our model on 1D Gaussian Processes regression and 2D moving MNIST/CelebA regression.
We apply ASNP to implement
Attentive Temporal-GQN and further evaluate on the moving CelebA task.
I NTRODUCTION
Neural Processes (NP) (Garnelo et al., 2018b) combine the strengths of neural networks and Gaussian processes (GP) such that, like Gaussian processes, it is ﬂexible in learning a new stochastic process at test time and also provides fast O (1) prediction speed like neural networks.
Learning from small datasets of multiple tasks (i.e., multiple stochastic processes), NP can be seen as a probabilistic latent variable framework for meta-learning.
Sequential Neural Processes (SNP) (Singh et al., 2019) extend the power of NP to a sequence of stochastic processes thus introducing a new class of sequential latent generative models.
It targets a large class of problems where the sequence of stochastic processes is governed by underlying transition dynamics and where learning to transfer information between stochastic processes is useful.
Despite these strengths, the remained question is whether NP and SNP can scale to more complex stochastic processes when the individual observations about the true process are highly partial, which prevails in many situations.
For example, an agent in a large complex landscape can only obtain very limited information from its immediate surroundings and it is desirable that it still maintains an accurate global scene representation that could also be changing with time.
Studying NP from this perspective, Kim et al.
(2019) demonstrates that the under-ﬁtting problem signiﬁcantly deteriorates the performance of NP.
To address this, ANP (Kim et al., 2019) and the method in Rosenbaum et al.
(2018) augment attention to NP and GQN (Eslami et al., 2018), respectively. Given the severe performance degradation of NP due to the under-ﬁtting, an important key question is whether SNP would also suffer from the under-ﬁtting problem or not , and its follow-up questions such as if so, how severely and in which settings would it happen? and how can we resolve the problem—would the same attention mechanism that worked for NP work for SNP as well?
In this paper, our goal is to answer these questions. We achieve this by not only showing that our proposed attention for SNP signiﬁcantly improves the standard SNP under the under-ﬁtting settings (see Fig. 1), but also by making a stronger claim i.e., in a sequential setting like ours, it is not optimal to perform attention on a memory buffer that simply stores all the observed contexts.
Instead, through our experiments, we claim that attention should be performed on a memory that is also updated (learned) at every time-step.
Consequently, a) this memory would learn to store an optimized representation of the past geared towards prediction and b) it would require fewer storage locations as it does not need to naively store every observed context points.
To implement such a memory, we introduce the concept of imaginary context that augments the set of real context points observed at any time-step.
The imaginary context contains a ﬁxed number of context points and is modeled as a random variable sampled at every time-step.
Given a query, we then apply attention on the union of the real and the imaginary contexts to generate predictions.
We call the proposed model as Attentive Sequential Neural Processes (ASNP).
In experiments on 1D regression tasks, we validate that SNP which simply attends on the memory buffer of all the past context points under-performs the proposed ASNP endowed with imaginary contexts.
Then in a comprehensive set of experiments on 1D and 2D regression and rendering tasks, we demonstrate ASNP’s performance gains over NP, ANP, SNP, GQN, and TGQN in different context regimes.
Our main contributions are as follows.
a) We show that SNP suffers from signiﬁcant under-ﬁtting by testing it in settings when the true stochastic process is complex (e.g., complex CelebA faces moving on a white canvas) or when each observation is highly partial (e.g., each context point is just one pixel).
b) We introduce a new model, Attentive Sequential Neural Processes (ASNP), that resolves the problem of under-ﬁtting by augmenting SNP with an attention mechanism.
To implement this, we introduce the novel concept of imaginary context .
We also empirically demonstrate that SNP which simply performs attention on a lossless memory of all the past context points still underperforms the proposed model and therefore the use of imaginary context is a better design choice.
c)
Using the attention paradigm introduced in ASNP, we extend TGQN to Attentive TGQN (ATGQN) for scene rendering.
d) We show that ASNP outperforms the baselines in various 1D and 2D tasks performing future prediction and meta-transfer learning i.e., making simultaneous use of the past knowledge and the current contexts.
B ACKGROUND
Neural Processes (NP) (Garnelo et al., 2018b) learns to learn a stochastic process P that maps an input x ∈ R
d x to a random variable y ∈ R d y .
It does so using a set of context observations
C =
( X C , Y C )
= { x i , y i } i ∈I ( C ) .
Here, I ( C ) is the set of indices for the elements in set C .
Speciﬁcally, to learn stochastic process from the context, NP uses an inference distribution P ( z | C ) and then uses the representation z ∼ P ( z | C ) of the inferred stochastic process to generate the prediction y given a query x .
This is modeled by p ( y | x, z ) .
The full generative process of NP conditioned on the context can be written as:
[Equation]
where P ( Y | X, z ) = (cid:81)
i ∈I ( D ) P ( y i | x i , z ) and D contains the target observations D =
( X, Y )
= { x i , y i } i ∈I ( D ) .
The training data for NP is obtained by drawing a stochastic process and then sampling ( C, D ) from the sampled stochastic process.
Treating each stochastic process as a task,
NP can be seen as a probabilistic meta-learning framework.
Generative Query Networks (GQN) (Eslami et al., 2018) uses modeling similar to NP for the problem of learning representation and rendering of 3D scenes from a set of partial 2D observations.
In GQN, a query x becomes a camera viewpoint in a 3D environment and an output y corresponds to an image taken from the viewpoint x .
The latent variable in the original GQN are query-dependent and may thus produce uncorrelated generations across queries.
Consistent GQN (CGQN) (Kumar et al., 2018) resolves this problem by having a single stochastic scene representation.
For simplicity, we use the term GQN to denote CGQN in the remainder of the paper.
Attentive Neural Processes (ANP) (Kim et al., 2019) resolves the problem of under-ﬁtting in NP.
To this end, ANP uses two context-encoding paths.
The ﬁrst path is the same as the scene encoder in
NP that summarizes all the context points agnostic to the target query x and produces a single vector of the global representation by average pooling.
The other path computes representations for each target query using query-dependent attention.
The set of representations returned r from the set of target queries X may be written as r = f attend ( X, X C , v C ) , where v C are the encoded representations of C via MLP or self-attention.
The attention function f attend is a key-value based attention which computes a response for each query q j ∈ X by ﬁrst computing its similarity with the key set
X C as w ji = f w ( k i , q j ) and then performing a weighted-sum on the value set i.e., s j = (cid:80)
i w ji v i , .
Here, the keys and values are the context queries x C and the representations v C , respectively.
The generative process is as follows: P ( Y | X, C )
= (cid:82) P ( Y | X, z, f attend ( X, X C , v C ))
P ( z | C )
d z. Kim et al.
(2019) propose ANP as a general model class which may be realized using different attention mechanism choices such as Dot-Product, Laplace or Multi-Head (Vaswani et al., 2017).
ANP needs more computation than NP but Kim et al.
(2019) mitigates this by showing faster training convergence than NP.
Sequential Neural Processes (SNP) (Singh et al., 2019) extend the capabilities of NP to model stochastic processes which may be decomposed as a sequence of stochastic processes P 1 , P 2 , · · · P T such that P t at time t depends on the past processes P <t through some underlying temporal dynamics.
At any t , P t is modeled as z t which depends on the current context C t and the past processes z <t using the transition model p ( z t | z <t , C t ) .
This enables us to use the general past trend as well as the current context to meta-transfer learn the process P t .
Given z t and a query x t , the prediction y t is generated using the observation model p ( y t | x t , z t ) .
In the SNP framework, at time-step t , we cannot directly access the past contexts, but we can still directly access the current context C t .
Thus, to model P t , we can use the stochastic encoding z t together with the direct encoding of C t .
Alternatively, this allows us to write the observation model as p ( y t | x t , z t , C t ) .
Combining the transition and the observation models, the complete generative process of SNP may be written as follows.
[Equation]
A TTENTIVE S EQUENTIAL N EURAL P ROCESSES
In this section, we describe the proposed Attentive Sequential Neural Processes (ASNP) that augments SNP with an attention mechanism.
3. G ENERATIVE P ROCESS
[Figure]
We recall that ANP uses the observation model p ( y | x, z, C ) which, based on the query x , performs attention on the context points in C .
In SNP, which uses the observation model p ( y t | x t , z t , C t ) , an analogous approach would use x t -dependent attention on C t .
However with this approach, attention is not effective because the current context is likely to be limited or even empty.
Instead, we want to perform attention not only on the current but also on the past contexts.
But to do that, we found that simply attending on a memory buffer of context history does not produce enough gains and leaves room for further improvement (see Appendix D.3). Instead, our hypothesis is that attention should be performed on a sequentially-updated memory so that the representations are optimized for prediction and are more size-efﬁcient.
To realize this, we propose augmenting the set of real context points C rt with a set of imaginary context points C it resulting in the full context set C t ≡ C r t ∪ C i t so that the observation model can now perform attention on this augmented C t .
Here, C it =
( X it , U it ) consists of imaginary queries (or keys)
X it =
{ x it,k } Kk =1 and its corresponding values U it = { u it,k } Kk =1 where K is a hyperparameter similar to the number of slots in a memory.
To generate C it , we introduce a novel and attentive transition model P ( C it | C <t , C rt ) treating C it as a random variable.
In this way, C it is a noisy reconstruction of the past sampled using an attention mechanism on the past contexts (real and imaginary) and on the real contexts of the current time-step.
Augmenting SNP with these, the generative model for ASNP may be written as follows.
[Equation]
Implementing the transition model P ( z t | z <t , C t ) .
With this model, we obtain the global representation z t which is a query-agnostic encoding of the current stochastic process.
This module can be considered similar to its counterpart in SNP without attention.
Note that it is a design choice whether to choose C t or C rt for the conditioning context of this module.
If we use C rt , z t becomes global context-encoding of real contexts.
Otherwise, it becomes an encoding of the combined contexts.
Implementing the context imagination P ( C it | C <t , C rt ) .
To implement this, we keep track of the
K imaginary context points at all time-steps and their generative model is as follows:
[Equation]
[Equation]
, C t )
For each imagination slot k , we ﬁrst sample the key x
it,k .
For this, we use an RNN that takes the past imaginary inputs
X it − 1 and the encoding of the real contexts v rt = f order-inv ( C rt ) to produce a hidden state h
Xt from which we then sample the imaginary key set X it .
Given these keys, the corresponding imaginary values u it,k are generated next.
For each imagination slot k , we maintain an imagination-tracker RNN that takes ( x it,k , u it − 1 ,k ) as input and produces a hidden state
h ut,k .
These hidden states act as intermediate imaginary values and we use them to construct a key-value set S it
= { ( x it,k , h ut,k ) } ∪ { ( x rt , f y → u ( x rt , y rt )) } .
On this set, we apply self-attention using only the imaginary x
it,k as attention queries and obtain the corresponding attention encodings a it,k .
Such encodings therefore also model the interactions among the context points.
From these, we sample the ﬁnal imagination values u it,k ∼ N ( f uµ ( a it,k ) , f uσ ( a it,k )) .
Implementing the observation model P ( y t | x t , z t , C t ) .
Taking x t as the attention query, we attend on C t , the union of the real C rt and imagined C it context, as the key-value set and obtain the attention encoding r x t = f attend ( x t , C t ) .
This is done after each real context x ∈ X rt and y ∈ Y rt is encoded using f y → u .
The attention encoding r x t is then concatenated with the global encoding z t to perform the prediction y t .
Algorithm 1 shows the described process of context imagination and encoding.
Discussion.
An interesting interpretation of the proposed modeling can be drawn in comparison to the memory retrieval process in the human brain (Eichenbaum, 2017).
As is well-known in neuroscience, “ human memory is not a literal reproduction of the past, but instead relies on constructive processes that are sometimes prone to error and distortion ” (Schacter, 2012).
Our model resembles this process in the sense that it (i) compresses the past observation and importantly what it has imagined in the past through an RNN (similarly to memory consolidation), and then after observing current context, (ii) recalls from the compressed memory not a simple copy of the past but a constructive recreation of representations that is optimized to help prediction as illustrated in Fig.
3. L EARNING AND I NFERENCE
Due to the intractability of the true posterior, ASNP is trained via variational approximation with the following posterior approximation:
[Equation]
where Q ( C i t | C i <t , C r , D ) =
Q ( U i t | X i t , C i <t , C r , D ) Q
( X i t | C i <t , C r , D ) and D =
( X, Y ) .
For training, the following evidence lower bound (ELBO) is maximized w.r.t.
θ and φ :
[Equation]
19 − 1 .−− 0 .− 0 .− 0 .0 .0 .
Time
T a r g e t N
L L 1DGPRegressionScenario(a)      19 − 1 .−− 0 .−
.− 0 .0 .0 .
Time 1DGPRegressionScenario(b) 1 6 11 16 21 26 31 36 41 46 −− 0 . 0 . 1 .
Time 1DGPRegressionScenario(c)
NPANPSNPASNP
For backpropagation, we use reparameterization trick (Kingma & Welling, 2013) for continuous variables.
Derivation of eq. 6 is provided in Appendix A.
R ELATED W ORKS
Learning to learn stochastic processes from few data points at test time has become attractive as meta-learning in recent times.
In this line of work, Conditional Neural Processes (CNP) (Garnelo et al., 2018a) models a regression function but without a global latent which causes inconsistency between values for different queries.
NP (Garnelo et al., 2018b) resolves this by proposing an explicit global latent path.
While CNP and NP are frameworks demonstrated on simpler tasks, GQN (Eslami et al., 2018) develops it to render 3D scenes.
Like CNP, GQN also produces inconsistency between queries due to the absence of a global latent.
CGQN (Kumar et al., 2018) resolves it by introducing a global scene latent.
To address the challenge of scalability to larger and more complex stochastic processes, ANP (Kim et al., 2019) identiﬁes and resolves the problem of under-ﬁtting using query-dependent attention.
Similarly, Rosenbaum et al.
(2018) introduces attention into the
GQN model to render complex 3D scenes in large procedurally-generated maps as in Minecraft.
To extend the model to stochastic processes decomposable as a sequence of stochastic processes,
SNP (Singh et al., 2019) was proposed.
It was then used to extend GQN to TGQN for rendering dynamic 3D scenes.
Functional Neural Processes (FNP) (Louizos et al., 2019) was proposed to learn a graph of dependencies between a pre-selected set of points and the training points for modeling distributions over functions.
The concept of imaginary context i.e. pseudo-context, which act as trainable representative points is related to the inducing points in Set Transformer (Lee et al., 2019) and sparse GPs (Snelson &
Ghahramani, 2006; Titsias, 2009).
VampPrior (Tomczak & Welling, 2017) also uses few trainable pseudo-inputs to resolve over-ﬁtting, over-regularization and high computational complexity.
The analogous reference set used in FNP to model a graph of small number of points is also motivated from the inducing points of sparse GPs.
With regard to the use of trainable memory, Differentiable
Neural Dictionary (Pritzel et al., 2017) stores and updates the previous trajectories as key-value pairs to quickly learn a variety of environments.
E XPERIMENTS
We evaluate ASNP on the following selection of tasks: a) 1D Gaussian Process (GP) regression b) 2D moving MNIST (LeCun et al., 1998) and 2D moving CelebA
(Liu et al., 2015).
On these tasks, we evaluate and compare the proposed model against NP, ANP, SNP as the baselines.
Each task is evaluated with three scenarios, which is used in (Singh et al., 2019) for 1D GP regression: (a) To evaluate the model’s ability
to extrapolate the future, we provide high-amount of context
19 − 1 .− 1 .−− 0 .− 0 .
Time
T a r g e t N
L L 2DMovingMNISTRegressionScenario(a)      19 − 1 .− 1 .−− 0 .− 0 .
Time 2DMovingMNISTRegressionScenario(b) 1 6 11 16 21 26 31 36 41− 1 .−− 0 .− 0 .− 0 .
Time 2DMovingMNISTRegressionScenario(c)
NPANPSNP
ASNP(k=9)ASNP(k=25)ASNP(k=100)      19 − 3 .−− 2 .−− 1 .−
Time
T a r g e t N
L L 2DMovingCelebARegressionScenario(a)
−− 2 .−− 1 .−
Time 2DMovingCelebARegressionScenario(b) 1 6 11 16 21 26 31 36 41 46 −− 2 .−− 1 .−
Time 2DMovingCelebARegressionScenario(c)
NPANPSNPASNP in the ﬁrst 10 time-steps out of the 20. (b) To test the model on its ability to track the dynamics using intermittently arriving context, we provide high-amount of context in 10 randomly chosen time-steps out of the 20. (c) To test the model’s ability to gather and make use of low amount of context received over long segments of time, we provide low amount of context in 45 randomly chosen time-steps out of the 50.
In each setting, the remaining time-steps are used for prediction.
The details about model architectures can be found in Appendix B.1.
To test the beneﬁts of attention for rendering 2D images, we used moving CelebA because it has higher uncertainty when given partial knowledge in comparison to an alternative data set such as moving simple shapes (e.g. circle).
We test on the scenario (c) as described above, but with shorter sequence lengths and smaller context sizes.
Details are explained in the section on 2d rendering.
1D Regression: We ﬁrst evaluated our method on sequential 1D Gaussian Process data set.
This is an extension of the GP data set through the addition of linear dynamics on the kernel hyperparameters as used in (Singh et al., 2019).
In each transition, we add a small Gaussian noise to introduce stochasticity.
For scenarios (a) and (b), the number of context observations n is randomly selected in [5 , 50] or an empty set.
The number of target observations m is randomly selected in [1 , 51 − n ] .
For scenario (c), n is 1 or 0, and m is chosen from [1 , 11 − n ] .
The target set subsumes the context.
The more details about this task is described in Appendix C.
In the quantitative results of Fig 3, SNP outperforms NP as reported in (Singh et al., 2019).
On the other hand, in scenario (a), the performance of ANP degrades steeply after the context are removed since it does not model the dynamics explicitly.
Even though it appears to outperform NP and SNP, this can be credited to the attention in ANP that prevents underﬁtting (Kim et al., 2019).
ASNP shows better performance for all the scenarios, outperforming ANP with a greater gap in scenario (c).
Further, it saturates the fastest among the tested methods for the entire scenarios as described in
Appendix D.1.
We hypothesize that when the dense context is given at each time-step in (a) and (b),
ANP can ﬁt well as it does not need to rely on the dynamics much.
But in scenario (c), the context is low in each time-step and the model must capture dynamics to make the best use of the context seen thus far.
In the qualitative results of Fig 3, we illustrate a sample of a validation set in scenario (c) with predictions from ASNP and the baselines.
As shown, even for the points not shown recently, ASNP predicts better than the baselines with lower uncertainty.
ANP and SNP fail to predict those points well.
This shows that the update of the imaginary queries and the representations is necessary over and above the sequential update of the global representations or the expensive modeling of the entire past using the time information.
More pieces of evidence about a useful role of imaginary context are shown in Appendix D.2 and D.3 where comparisons with larger size baselines and ASNP without imaginary context are described and ASNP outperforms them.
It also shows a having more parameters in baselines is not sufﬁcient and imaginary context plays a signiﬁcant role.
More qualitative results are in Appendix E.1.
2D Regression: We take a step further to evaluate our model on a more challenging 2D setting where the queries are the pixel locations and the regression is over the pixel values of images.
The image is a 42 × 42 -sized black canvas with a moving MNIST digit (sized 28 × 28 ) or a cropped CelebA face (resized to 32 × 32 ).
The digits or the faces start moving from a random location heading towards a direction randomly selected at t = 0 with a speed of 3 pixels per time-step and bouncing perfectly upon encountering a wall.
Like in the regression task, a small Gaussian noise is added to the location obtained after each transition.
Same as the regression task, 3 context regimes are designed.
In regime (a) and (b), n is either 0 or n ∈ [5 , 500] while m is drawn from [1 , 501 − n ] .
In (c), n is either 0 or 30 and m is drawn from [1 , 51 − n ] .
The target set subsumes the context.
We present the quantitative results for 2D regression tasks in Fig.
From these, we infer the following points.
We observe that SNP performs better than ANP except in the early time-steps of scenario (a) when a large number of contexts points are shown without interruptions.
We reason that in the presence of such ample context points, NP and SNP still suffer from under-ﬁtting while
ANP can make an effective use of those points without having to model the dynamics.
But when the task setting demands that the model capture the dynamics, as in task (c), both NP and ANP perform poorer than SNP and ASNP.
This is more pronounced in the 2D setting than in the 1D because tracking dynamics of images without explicit temporal modeling is harder.
We further note that
MNIST digits are simpler than the CelebA faces.
Therefore in the latter case, SNP struggles with underﬁtting and ANP struggles with modeling the dynamics, causing them both to furnish similar performances in task (b).
ASNP does not suffer from either of these problems and it outperforms all baselines in all the tasks.
In scenario (a), we also note a sudden performance drop in NP, ANP and ASNP.
While unsurprising in NP and ANP, it also occurs in ASNP due to sudden switch-over to a purely imaginary context.
This drop does not appear in the 1D tasks since the number of context points provided are fewer and therefore ASNP is less impacted by their absence.
We also evaluate the effect of varying the imaginary context sizes on the moving MNIST data set for the scenarios (a) and (b).
We observe that a larger k yields better performance.
This gain, however, saturates quickly as the improvement between k = 9 and k = 100 is not large.
This shows that the imaginary context can represent the stochastic process with only a few points. [Figure] [Figure] [Figure]
e t N [Figure] In Fig.
5, we present the qualitative results on the two data sets for scenario (c).
Here, we note that
NP and ANP can produce the digit location but not the appearance.
SNP captures the appearance of the digit but not clearly.
Although ASNP also produces fuzzy appearance in the early time-steps, the digit becomes identiﬁable by t = 10 and the details keep getting clearer over t = 20 .
We also see a similar trend for the moving CelebA task.
More illustrations of scenarios (a), (b) and (c) are included in Appendix E.2.
2D Scene Rendering.
We also introduce the attention paradigm of ASNP into Temporal
GQN (Singh et al., 2019) and test it on the moving CelebA rendering task.
We call it Attentive
TGQN (ATGQN).
To design this, we use the Temporal-ConvDRAW as in TGQN (Singh et al., 2019) for encoding a global latent.
For computing a query-dependent representation, we apply the
ASNP’s attention encoder with imagination to get a scene-wise matrix.
We use the same decoder as in TGQN.
While it is a limited solution working on the task where y t can be divided as points with queries, it resolves the underﬁtting for overlapping pixels between the context and the target.
Unlike
NPs, partial overlap exists, which is difﬁcult to predict by scene-wise attention.
More details about the model architecture is described in Appendix B.2.
The canvas size of the data set is 80 × 80 and cropped face image size is resized in 64 × 64 .
The direction of motion and initial position are randomly selected at t = 0 with 13 pixels per time-step speed.
The size of output y is 32 × 32 .
The sequence length is 25.
One context view is provided to the model at each of the randomly chosen 20 time-steps out of 25 time-steps.
and an empty set is given on the remaining 5 time-steps.
The target size m is drawn in the range [1 , 11 − n ] .
In Fig.
6, we show quantitative and qualitative results for scenario (c).
We can see that ATGQN outperforms TGQN because for the overlapping areas, ATGQN resolves the underﬁtting.
The performance gap between TGQN and ATGQN, however, is smaller compared to the previously seen 1D and 2D regression tasks.
A similar trend is observed in scenario (a) whose results are in Appendix
D.4.
The reason for this is low uncertainty in the presence of the context that provides a wide view of the canvas.
This and the small number of context causes lesser underﬁtting than what would have arisen had the context observations been highly partial as in the regression tasks.
More examples are included in Appendix E.3.
C ONCLUSION
In this paper, we addressed the problem of under-ﬁtting that affects Neural Processes and Sequential
Neural Processes.
Although this is resolved in the former by ANP, it is not possible to resolve it in SNP with direct application of attention since SNP cannot store the past context.
We introduced
Attentive Sequential Neural Processes which comprises a novel memory mechanism of imaginary context to resolve the under-ﬁtting.
It not only compresses the past knowledge but does it in a way that is geared towards making good predictions with low uncertainty.
In the experiments, the proposed model shows superior performance on various tasks for a number of sequential scenarios.
Since ASNP is a model based on attention, its limitations include those of any attention-based model.
One such case is the scene-wise attention.
Scene-wise attention cannot ﬁt properly for partial overlapped region between context and target.
Although we partially solve it with pixel-wise attention, it would still be interesting to encode query dependent representation for scene observation.
A PPENDIX A ELBO D ERIVATIONS
[Equation]
where
Q φ ( z t , C it | C r , D ) =
(cid:81) tt (cid:48)
=1 Q ( z t (cid:48)
| z <t (cid:48) , C it (cid:48) , C r , D ) Q ( C it (cid:48)
| C i<t (cid:48) ,
C r , D ) and
Q φ ( z <t , C i<t ) = (cid:81) t − 1 t (cid:48)
=1 Q ( z t (cid:48) , C it (cid:48)
| z <t (cid:48) , C i<t (cid:48) , C r , D ) for simplicity.
[Figure] [Figure]
Every models share a basic architecture (e.g. context encoder and decoder).
They have two encoders.
One of them is for a global latent.
It consists of 3 layers MLP with ReLU (Nair & Hinton, 2010) is used as encoder and 2 layers MLP is used to encode and sample a latent.
Another encoder is consisted of 6 layers MLP with ReLU activation function and mean function for NP and SNP and attention module for ANP and ASNP.
Decoder is consisted of 4 layers MLP with ReLU activation function and 1 layer MLP is used to sample ˆ y with uncertainty.
For non sequential models (NP and ANP), time is encoded as a normalized ﬂoat scalar, e t = 0 .
25 + 0 .
5( t/T ) , where T is the length of sequence of data.
After encoding, it is appended in query as x (cid:48) =
( x, e t ) .
For attention models (ANP and ASNP), Multihead attention (Vaswani et al., 2017) is used because it showed the best performance in a variety of attention methods (Kim et al., 2019).
ASNP has two attention modules, which shares parameters.
For sequential models (SNP and ASNP), they uses same temporal architecture for a global latent.
LSTM with a default setting of TensorFlow (Abadi et al., 2016) is used and the dimension of hidden unit of LSTM is d r where d r is the dimension of representations.
For ASNP, two RNN modules for the imaginary queries and representations is LSTM with a default setting of Tensorﬂow.
The dimension of hidden unit of LSTMs is d r .
Here, the number of state of LSTM for representations is k × batch size.
The dimension of representation d r is 128 and the dimension of query d q is 1 for 1D regression and 2 for 2D regression.
Note that for non-sequential model, d q is 2 and 3 by attaching encoded time e t .
The number of imaginary context is 25.
The initial imaginary context C i 0 is trainable parameters.
Learning rate is 0.0001 and batch size is 16 for 1D regression and 8 for 2D regression.
The diagram for ASNP architecture is in Fig.
B. GQN S
For an encoder, tower representation (Eslami et al., 2018) is used.
A decoder is similar between models (GQN, TGQN and ATGQN) based on the decoder of CGQN (Kumar et al., 2018) that is a deterministic generative model working on auto-regressive manner.
The different is inputs that are described in bellows.
For GQN, time is encoded and appended in query same to NP and ANP.
To sample z t , we apply a convolutional DRAW(ConvDRAW)
(Gregor et al., 2015; 2016) like GQN (Kumar et al., 2018).
To render ˆ y , the decoder inputs z t , X t and v rt that is an output of the decoder.
v rt is averaged representation that is an input for every target queries.
For TGQN, Temporal-ConvDRAW (Singh et al., 2019) is applied to sample z t with ConvLSTM (Xingjian et al., 2015) state, h t .
We take RSSM to implement as (Singh et al., 2019).
The decoder input is a set of the global latent z t , the target queries X t and ConvLSTM state h t , in which, z t and h t are independent to target queries.
For ATGQN, a global latent is sampled same to TGQN and a query dependent representation of
ASNP is used.
To make a pixel-wise path, we split scene view as pixel view with queries per each pixel.
When many context are given, overlapped between context exists.
We leave it to optimize batch operation.
We encode the pixel-wise representation with 2 convolutional layers to make same dimension with scene-wise, the dimension of which is [ scene-height / 4 , scene-height / 4 , 3] .
The input of decoder are z t , X t and r t = { r ti } i ∈I ( D ) that is query dependent representations for each target query x ti .
The dimension of representation d r is 128.
The number of steps for (Temporal) ConvDRAW to sample z t is 3 and the dimension of z is 4.
Note that z t is the output of the last roll-out of (Temporal) ConvDRAW.
For TGQN and ATGQN, the number of hidden unit for ConvLSTM is 40.
The dimension of hidden unit on the decoder is 32 and the number of steps for auto-regressive decoding is 6 and output is cultivated.
The query size is 2 (for GQN, it is 3 with e t ).
The number of pseudo context is 100.
Learning rate and batch size is 0.0001 and 4.
A PPENDIX C G AUSSIAN P ROCESS D ATA S ET
The kernel hyper-parameters, length-scale l and kernel-scale σ are selected randomly at t = 0 in [0 . 7 , 1 . 2] and [1 . 0 , 1 . 6] for scenarios (a) and (b).
For scenario (c), l and σ are chosen in [1 . 2 , 1 . 9] and [1 . 6 , 3 . 1] at t = 0 .
The true underlying dynamics of the kernel hyper-parameters ∆ l and ∆ σ are in [ − 0 .
03] and [ − 0 .
05 , 0 . 05] chosen at t = 0 .
Gaussian noise on the dynamics is ∼ N (0 , 0 . 1) .
A PPENDIX D A DDITIONAL E XPERIMENTAL R ESULTS D. L EARNING C URVE ON GP TASK
[Figure]
(inhundredthousands)
T a r g e t N
LL 1DGPregressiontask(a)
[Figure] [Figure]
[Figure]
−− 0 .− 0 .− 0 .
[Figure]
[Figure] [Figure]
[Figure]
Iterations(inhundredthousands) 1DGPregressiontask(b)
[Figure]
9 · 10−− 0 . 0 . 1 .
Iterations(inhundredthousands) 1DGPregressiontask(c)
NPANPSNPASNP     5 · 10− 1 .−− 0 .− 0 .− 0 .0 .0 .
Seconds(inhundredthousands)
T a r g e t N
LL 1DGPregressiontask(a)
5 · 10− 1 .−− 0 .− 0 .− 0 .0 .0 .
Seconds(inhundredthousands) 1DGPregressiontask(b) 0 .0 .0 .0 .
· 10−− 0 . 0 .
Seconds(inmillions) 1DGPregressiontask(c)
NP ANP
SNPASNP
In this section, we show target NLL for iterations and wall clock time (Figs.
8) for 1D GP data set.
ASNP and ANP computation complexity is bigger than NP O (1) due to attention.
ANP needs
O ( m (cid:80)
Tt n t ) where n t is the context size at time-step t .
Because it needs to attention to entire context in current.
ASNP computation complexity is O (( n + k ) m ) for generating r t and O (( n + k ) k ) for updating the pseudo context.
When the sequence of data T is not long, ANP computation complexity is smaller than the complexity of ASNP.
However, the learning curve for wall clock time shows ASNP quickly saturates on lower loss in same time.
D. E VALUATION ABOUT BASELINES WITH VARIOUS PARAMETER SIZES FOR GP TASK
19 − 1 .−− 0 .− 0 .− 0 .
[Figure]
Time
T a r g e t N
LL
[Figure]
In this section, we evaluate our model with baselines implemented with a larger size of neural networks.
The representation dimension of baselines (i.e., NP, ANP, and SNP) are selected as d r = 512 .
This is chosen as the performance improvement saturates beyond this value in (Kim et al., 2019).
The remaining settings are the same as described in the previous sections.
Fig.
9 shows that NP and SNP implemented with larger network perform better than their smaller sized counterparts.
On the other hand, larger sized ANP shows a similar performance to its smaller counterpart.
We thus hypothesize that the smaller model capacity is enough to train ANP on this task.
ASNP, although implemented with a smaller network, still clearly outperforms the rest in scenario (c) even against baselines of larger sizes and this gap gets bigger as the time goes on.
It shows that the baselines having a bigger network is not sufﬁcient to represent a sequence of stochastic processes and temporal encoding of ASNP and the imaginary context plays a signiﬁcant role.
D. ASNP WITHOUT I MAGINARY C ONTEXT
[Figure]
a r g e t N
LL
[Figure]
In this section, we evaluate the effect of imaginary context on the ASNP performance.
To validate it, we design ASNP that does not produce imaginary contexts but simply stores all the real contexts from K most recent time-steps.
We call this model as ASNPK in this section.
To validate not just the effect of the imaginary context but also the past knowledge on prediction, we test with K = 1, 3, 5 and inﬁnity.
Here, when K = inﬁnity, ASNPK stores the entire set of the past contexts like
ANP and the number of context to attend of ASNPK is usually bigger than the number of imaginary context per each time-step (i.e. if K = 3 , it could be up to 150 contexts for scenario (a) and (b) while
ASNP attends up to 75 .).
ASNPK has two RNN modules for the global latent and the deterministic variable, but it attends contexts from multiple time-steps.
To optimize the attention performance of
ASNPK, we also encode the time information into the queries like NP and ANP.
In Fig.
10, quantitative results for the GP tasks are plotted.
As K increases from 1 to inﬁnity, ASNPK shows performance enhancement which quickly saturates.
However, it still under-performs with respect to ANP and ASNP.
For ANP, attending all the past knowledge (ANP) makes better performance than what using some past context (ASNPKs except K =inﬁnity).
ASNPK with
K = inﬁnity shows similar performance to ANP because ASNPK(inf) tightly depends on more informative attention than temporal encoding.
For ASNP, the simple copy of the past cannot be better than the optimized learned representation of ASNP imaginary contexts.
It shows that a sequential memory update mechanism like our brain is more effective than attention on the lossless copies of real past contexts.
D. 2D RENDERING TASK FOR SCENARIO ( A )
We also test our model on the moving CelebA rendering task for scenario (a).
The sequence length
T sets as 15 and the context is given to t = 7 .
The context size n is randomly selected in [1 , 10] or 0.
The target size m is randomly chosen in [1 , 11 − n ] .
Other environment setting is same to scenario (c).
[Figure]
Fig.
11 shows results that are both quantitative and qualitative.
Target NLL value of GQN is very high.
On the other hand, the generation performance is not the same as the target NLL.
The explanation for this is inability to predict the complex face image’s location.
When context is given,
ATGQN outperforms TGQN because ATGQN solves the overlapping region underﬁtting.
ATGQN however shows similar outcomes to TGQN on non-context prediction.
The explanation for this is less uncertainty due to a lot of contextual information.
[Figure]
[Figure] [Figure]