Paper:
PERCEPTION UPDATING NETWORKS: ON ARCHITEC- TURAL CONSTRAINTS FOR INTERPRETABLE VIDEO GENERATIVE MODELS
We investigate a neural network architecture and statistical framework that models frames in videos using principles inspired by computer graphics pipelines. The proposed model explicitly represents “sprites” or its percepts inferred from maximum likelihood of the scene and infers its movement independently of its content. We impose architectural constraints that forces resulting architecture to behave as a recurrent what-where prediction network.
1 INTRODUCTION
The current computer graphics pipelines are the result of efficient implementations required by limited hardware and high frequency output requirements. These requirements were also achieved with the use of explicit physics and optic constraints and modeling with constantly improving data structures (Shirley et al., 2015).
In machine learning on the other hand, for a long time image (Olshausen et al., 1996) and video (Hurri & Hyvärinen, 2003) generative models had been investigated with statistical approaches that model images down to the pixel level (Simoncelli & Olshausen, 2001), sometimes assuming neighborhood statistical dependencies (Osindero & Hinton, 2008). In video prediction, the current state of the art uses variations of deep convolutional recurrent neural networks (Kalchbrenner et al., 2016) (Lotter et al., 2016) (Finn et al., 2016).
As a parallel to the classic machine learning approach to image and video interpretation and prediction is a growing trend in the deep learning literature for modeling vision as inverse graphics (Kulkarni et al., 2015)(Rezende et al., 2016)(Eslami et al., 2016). These approaches can be interpreted into two groups: supervised and unsupervised vision as inverse graphics. The supervised approach assumes that during training an image is provided with extra information about its rotation, translation, illumination, etc. The goal of the supervised model is to learn an auto-encoder that explicitly factors out the content of the image and its physical properties. The supervised approach is illustrated by Kulkarni et al. (2015).
The unsupervised approach requires extra architectural constraints, similar to those assumed in computer graphics. For example, Reed et al. (2016) modeled the content of a scene with a Generative Adversarial Network (Goodfellow et al., 2014) and its location with Spatial Transformer Networks (Jaderberg et al., 2015). The full model is adapted end-to-end to generate images whose appearance can be changed by independently modifying the ”what” and/or ”where” variables. A similar approach was applied to video generation with volumetric convolutional neural networks (Vondrick et al., 2016).In two papers by Google DeepMind (Rezende et al., 2016) (Eslami et al., 2016) they improved the ”where” representations of the unsupervised approach and modeled the 3D geometry of the scene. This way they explicitly represented object rotation, translation, camera pose, etc. Their approaches were also trained end-to-end with REINFORCE-like stochastic gradients to backpropagate through non-differentiable parts of the graphics pipeline (Rezende et al., 2016) or to count
∗Companion code repo coming soon.
the number of objects in the scene (Eslami et al., 2016). Those papers also used Spatial Transformer Networks to model the position of the objects in the scene, but they extended it to 3D geometry so it could also model rotation and translation in a volumetric space.
Other approaches inspired by the graphics pipeline and computer vision geometry in machine learning uses the physics constraints to estimate the depth of each pixel in the scene and camera pose movements to predict frames in video (Mahjourian et al., 2016) (Godard et al., 2016).
The present paper is closer to the unsupervised approach of vision as inverse graphics. More precisely, here we investigate frame prediction in video. Contrary to the work by Reed et al. (2016) here we first limit ourselves to simple synthetic 2D datasets and learning models whose representations can be visually interpreted. This way we can investigate exactly what the neural network is learning and validate our statistical assumptions. Also, we investigate the behavior of Spatial Transformer Networks and question it as the default choice when limited compute resources are available and no scale invariance is required.
First in the next Section we will pose a statistical model that is appropriate for machine learning but inspired by the graphics pipeline.
2 A 2D STATISTICAL GRAPHICS PIPELINE
This section starts with a high level description of the 2D graphics pipeline, followed by a discussion of how to implement it with neural network modules, and finally we define a formal statistical model.
The 2D graphics pipeline starts from geometric primitives and follows with modeling transformations, clipping, viewing transformations and finally scan conversion for generating an image. Here, we will deal with previously rasterized bitmaps, i.e. sprites, and will model the translation transformations, rotation and clipping with differential operations. This way, the steps in the pipeline can be defined as layers of a neural network and the free parameters can be optimized with backpropagation.
For our neural network implementation, we assume a finite set of sprites (later we generalize it to infinite sprites) that will be part of the frames in the video. The image generation network selects a sprite, s, from a memorized sprite database Si∈{1,...,K} using an addressing signal c:
s = ∑ j
cjSj , where∑ j cj = 1. (1)
For interpretable results it would be optimal to do one-hot memory addressing where cj = 1 for Sj = S and cj = 0 otherwise. Note that (1) is differentiable w.r.t to both cj and Sj so we can learn the individual sprites from data. We can for cj to sum to 1 using the softmax nonlinearity. This approach was inspired by the recent deep learning literature on attention modules (Bahdanau et al., 2014) (Graves et al., 2014).
When the number of possible sprites is too large it is more efficient to do a compressed representation. Instead of using an address value c we use a content addressable memory where the image generator estimates a code z that is then decoded to the desired sprite with a (possibly nonlinear) function d(z). If we interpret the addressing value z as a latent representation and the content addressable memory d(z) as a decoder, we can use the recent advances in neural networks for generative models to setup our statistical model. We will revisit this later in this section.
The translation transformation can be modeled with a convolution with a Delta function or using spatial transformers. Note that the translation of an image I(x, y) can be defined as
I(x− τx, y − τy) = I(x, y) ? δ(x− τx, y − τy), (2)
where ? denotes the image convolution operation. Clipping is naturally handled in such a case. If the output images have finite dimensions and δ(x−τx, y−τy) is non-zero near its border, the translated image I(x− τx, y − τy) will be clipped. Another way of implementing the translation operation is using Spatial Transformer Networks (STN) (Jaderberg et al., 2015). An implementation of STN can be defined in two steps: resampling and bilinear interpolation. Resampling is defined by moving the position of the pixels (x, y) in the original image using a linear transform to new positions (x̃, ỹ) as
[ x̃ ỹ ] = A [ x y 1 ] , where
A = [ A11 A12 A13 A21 A22 A23 ] .
(3)
We assume the coordinates in the original image are integers 0 ≤ x < M and 0 ≤ y < N , where M ×N is the size of the image I . Once the new coordinates are defined, we can calculate the values of the pixels in the new image Ĩ using bilinear interpolation:
Ĩ(x̃, ỹ) = wx1,y1I(x1, y1) + wx1,y2I(x1, y2)+
wx2,y1I(x2, y1) + wx2,y2I(x2, y2) (4)
where (x1, x2, y1, y2) are integers, x1 ≤ x̃ < x2, y1 ≤ ỹ < y2 and
wx1,y1 = (bx̃c − x̃)(bỹc − x̃) wx1,y2 = (bx̃c − x̃)(bỹc+ 1− ỹ) wx2,y1 = (bx̃c+ 1− x̃)(bỹc − ỹ) wx2,y2 = (bx̃c − x̃)(bỹc+ 1− ỹ)
(5)
To avoid sampling from outside the image we clip the values bx̃c and bx̃c+1 between 0 and M and the values bỹc and bỹc + 1 between 0 and N . We omitted that in (5) for conciseness. Note that (4) is piecewise differentiable w.r.t I .
We can define translation through operations with
A = [ 1 0 τx 0 1 τy ] . (6)
Also, we can rotate the image ρ radians counter clockwise with
A = [ cos ρ sin ρ 0 − sin ρ cosρ 0 ] . (7)
Image rescaling is achieved on that framework by rescaling in the right square submatrix A1:2,1:2. We illustrate in Fig. 1 how to get similar results using convolutions with a delta-function and spatial transformers.
Considering the tools defined above, we can define a statistical model of 2D images the explicitly represents sprites and their positions in the scene. We can use the free energy of this statistical model to optimize a neural network. Let us start with a static single frame model and later generalize it to video.
Let an image I ∼ pθ(I) be composed of sprite s ∼ pθ(s) centered in the (x, y) coordinates in the larger image I . Denote these coordinates as a random variable δxy ∼ pθ, where θ are the model parameters. pθ(δxy) can be factored in two marginal categorical distributions Cat(δx) and Cat(δy) that models the probability of each coordinate of the sprite independently. For the finite sprite dataset, pθ(s) is also a categorical distribution conditioned on the true sprites. For this finite case the generative model can be factored as
pθ(I, s, δ) = pθ(s)pθ(δxy)p(I|s, δxy), (8)
assuming that “what”, s, and “where”, δxy , are statistically independent. Also, in such case the posterior
pθ(s, δ|I) = pθ(s|I)p(δxy|I) (9)
is tractable. One could use for instance Expectation-Maximization or greedy approaches like Matching Pursuit to alternate between the search for the position and fitting the best matching shape. For the infinite number of sprites case, we assume that there is a hidden variable z from which the sprites are generated as p(s, z) = pθ(z)pθ(s|z). In such case our full posterior becomes
pθ(z, s, δ|I) = pθ(z, s|I)p(δxy|I) = pθ(z|I)pθ(s|I, z)p(δxy|I).
(10)
We can simplify (10) assuming pθ(z|s) = pθ(z|I) for simple images without ambiguity and no sprite occlusion. For a scalable inference in the case of unknown θ and z and intractable pθ(z|s) we can use the auto-encoding variational Bayes (VAE) approach proposed by Kingma & Welling (2013). Using VAE we define an approximate recognition model qφ(z|s). In such case, the loglikelihood of the i.i.d images I is log pθ(I1, . . . , IT ) = ∑T i log pθ(Ii) and
log pθ(Ii) = DKL(qφ(z|si)||pθ(z|si))+ DKL(pθ(z|si)||pθ(z|Ii))+
L(θ, φ, δxy, Ii). (11)
Again, assume that the approximation pθ(z|s) = pθ(z|I) we have DKL(pθ(z|si)||pθ(z|Ii)) = 0 and the free energy (or variational lower bound) term equal to
L(θ, φ, δ, I) = −DKL(qφ(z|si)||pθ(z))+ Eqφ(z|s,δ)pθ(δ|I)[log pθ(I|z, δ)],
(12)
where we dropped the subindices xy and i to avoid clutter. Here we would like to train our model by maximizing the lower bound (12), again inspired by VAE. We can do so using the reparametrization trick assuming qφ(z|s) and the prior pθ(z) to be Gaussian and sampling
z = mφ(I) + vφ(I) · ξ, (13)
where ξ ∼ N (0, σI), I is the identity matrix, the functions m(I) and v(I) are deep neural networks learned from data.
One can argue that given z and a good approximation to the posterior qφ, estimating δ is still tractable. Nevertheless, we preemptively avoid Expectation-Maximization or other search approaches and use instead neural network layers lx and ly:
δxy = softmax(lx(I))⊗ softmax(ly(I)), (14)
with ⊗ denoting the outer product of marginals. We also experiment using STNs. Such amortized inference is also faster in training and test time than EM and will also cover the case where I is itself a learned low dimensional or latent representation instead of an observable image. Bear this in mind while we use this approach even in simple experiments such as those with moving shapes in the Experiments Section. This will help us to understand what can be learned from this model.
We extend the model above to videos, i.e. sequences of images I(t) = {I(0), I(1), . . .}, assuming that the conditional log-likelihood log pθ(It|HIt) = logpθ(It|Hδt , Hzt) follows (11), where HIt is the history of video frames prior to time point t. Also Hδt and Hzt are the history of position coordinates and the history of latent variables of the sprites respectively. We should observe that one can make the assumption that the sprites don’t change for a given video I(t) and only estimate one sprite st=0 or hidden variable zt=0. This assumption can be useful for long term predictions, but requires that the main object moving in the scene doesn’t change.
In the next section, we propose a neural network architecture for maximizing our approximate variational lower bound 2D videos.
3 PERCEPTION UPDATING NETWORKS
This Section proposes a family of neural architectures for optimizing the lower bound (12). A schematic diagram is represented in Fig. (2). The core of our method is a Recurrent Neural Network (RNN) augmented with task specific modules, namely a sprite addressable memory and modeling transformations layers. RNNs augmented with task specific units were popularized by Graves et al. (2014) in the context of learning simple differentiable algorithms and served as inspiration for us as well. Here since we explicitly model the perceived sprites as s or z and update it and its location and/or rotation though time we decided to call our method simply Perception Updating Networks.
Here an input frame at time t, It, is fed to the RNN that emits 2 signals: a memory address that selects a relevant sprite and transformation parameters. If we are doing the translation transformation using convolutions and delta functions this output is equal to (14). If using STN, the translation operation returns the matrix A used in (3). Note that we could use both, letting convolutions with δ to the translation is constraining A as in (7) to do rotation transformations only. We describe the general case where both δxy and STNs are used in Algorithm 1.
Beyond deciding between STNs vs δxy , a few other free parameters of our method are the type of RNN (e.g. vanilla RNN, LSTM, GRU, ConvRNN, etc), the number of neurons in the hidden state of the RNN and neural network architectures that infer the correct sprite and modeling transformation parameters. Our hyperparameter choices are investigated separately in each experiment in the next Section.
Data: input videos It, t ∈ {0, 1, 2, . . .}, initial RNN state h0, neural network layers mφ, vφ, d, l, f Result: video predictions It, t ∈ {1, 2, 3, . . .} for t ∈ {0, 1, . . .} do
ht ← RNN(It, ht−1) δxy = softmax(lx(ht))⊗ softmax(ly(ht)) ρ = f(ht)
A = [ cos ρ sin ρ 0 − sin ρ cos ρ 0 ] ξ ∼ pθ(z) zt = mφ(ht) + vφ(ht) · ξ st = d(zt) at = STN(st, A)
Ĩt+1 = at ? δxy It+1 = µĨt+1 + (1− µ)B
end Algorithm 1: Perception Updating Networks. STN denotes spatial transformer operator (3)-(4) and ? denotes convolution. We experimented with several variations of this algorithm, mainly changing if and how the “where” modules δxy and STN are used. Also changing how the sprite st is calculated and not using a background B when not necessary.
In the next section we present experiments with the proposed architecture on synthetic datasets.
4 EXPERIMENTS
In this section we experiment with several implementations of the proposed Perception Updating Networks. We start with a simple synthetic dataset made of videos where one of 3 moving shapes moves with constant speed bouncing in the edges of an image. This illustrates the working of the finite memory and the addressing scheme in (1). Afterwards we show results on the moving MNIST dataset (Srivastava et al., 2015) commonly used in the literature of generative neural network models of videos.
4.1 BOUNCING SHAPES
In this first experiment we generate videos of one of three shapes moving on a non-zero background. The shapes are a square, triangle and cross. The image size is 20×20 pixels and the shapes are 8×8 pixels. The pixel values are between 0 and 1. The shapes are picked with equal probability and they move at constant speed of 1 pixel per frame. The shapes start from random initial positions with and start moving in random directions as well.
We tested two implementations of the proposed architecture: one using only convolutions, referred to as convolutional PUN in the figures, and another using using spatial transformers, called spatial transformer PUN. For the parameters of the convolutional PUN the RNN used was a Long Short Term Memory (LSTM) with 100 cells. The RNN in the Spatial Transformer PUN had 256 cells. In the convolutional PUN, the location layers used to calculate δxy , lx and ly , output vectors of size 20 pixels and we used the finite addressable memory described in (1). The background is also learned from data as weights of neural network. This background served to make the task more difficult and force the network to avoid just exploiting any non-zero value. After the convolutional composition It = st ? δxy , we added the background to form a new image using Ĩt = µ · It + (1− µ)B, where µ is a differentiable mask that accounts for the “transparency” of the image It. B is the learned 20 × 20 pixels background image. For complex shapes this mask shape could be calculated as another module in the network, similarly to the approach in Vondrick et al. (2016).
In the following experiments, the training videos were 10 frames long. At test time the network is fed the first 10 frames of a video and asked to predict the next 10. Results for the compared methods are shown in Fig. ??. For the baseline method, we did a hyperparameter search on conventional LSTMs with a single linear output layer until we found one that had comparable results at test time. That network had 256 hidden cells. Also, note that although the scale of the mean square error is the same, the results from our proposed architecture look smoother than those learned by the LSTM as shown in Fig. 3.
Given such a simple experiment, it is elucidating to visualize values learned by each piece of the network. As expected the sprite memory learned the 3 investigated shapes in transposed order since they are reverted by the convolution operation to compose the frame. We also experimented with choosing the size of the learned sprites st smaller and larger than the true shapes. We observed that for larger shapes such as 10 × 10 the sprites converge to the correct shapes but just using part of the pixels. For smaller shapes such as 6 × 6 pixels, instead of learning a part of the correct shape, the convolutional Perception Updating Network learned to compensate for the lack of enough pixels with more than one non-zero value in the location operation δxy (see Fig. 3). This allow us to suggest to the interested practitioner that in order to get interpretable results it is better to use sprites larger than the expected size than smaller.
For the spatial transformer PUN the image is calculated as (see Algorithm 1 for context):
A = f(ht), It+1 = STN(st, A). (15)
We noticed that the spatial transformer PUN was not able to learn the training videos using an equivalent architecture to the convolutional PUN one. We had to use multiple layers to define the function f(ht). In other words, in the convolution based method δxy can be estimated by a single affine transformation of the state ht but A cannot. We also had to use smaller learning rates to
guarantee convergence: 0.0001 for STN while the δxy-based model worked with a value 10 times larger.
If we don’t use the softmax nonlinearity to construct δxy the representations learned by the convolutional PUN are no longer visually interpretable. It is interesting to conclude that under this framework the “what” and “where” can only be distinguished if we impose architectural constraints. The reason is the commutative property of the convolution operation.
As a note on rotation, we ran experiments where the sprite are rotated by a random angle before being placed in the image. This new type of videos cannot be learned using only convolutional based Perception Updating Networks unless we increase the number of sprites proportionally to the number of possible angles. Spatial transformer based Perception Updating Networks can handle this new type of video naturally. Nevertheless, if the number of rotation angles is finite or can be discretized we found that we could learn to generate the videos faster if we combined the convolutional approach with a mechanism to select the appropriate angle from a set of possibilities. Results on this experiment are not shown in this paper due to space constraints but they can be reproduced with the companion code.
4.2 MOVING MNIST
The Moving MNIST benchmark uses videos generated by moving 28×28 pixel images of hand written digits in a 64× 64 pixels canvas. Just like in the Bouncing Shapes dataset, the digits move with different different speeds in different directions and can bounce in the walls. Unlike the Bouncing Shapes dataset, there are 60000 different sprites for training and 10000 for test, making it impractical to use a discrete memory module. Instead, we use the memory representation denoted by (13) followed by st = d(zt) as written in Algorithm 1.
We trained a convolutional Perception Updating Network using 2 layer LSTMs each one with 1024 cells for 200 epochs, with 10000 gradient updates per epoch. The latent variable z had 100 dimensions and the decoder d(·) was a single hidden layer MLP with 1000 hidden neurons and softplus
activation function. The output layer of this MLP has 784 neurons, which is the size of an MNIST image, and sigmoid activation function. In the test set we obtained a negative log-likelihood of 239 nats with the proposed architecture, while a 2 layer LSTM baseline had 250 nats. Note that the our method was optimized to minimize the lower bound (12), not only the negative likelihood. These results are not as good as those obtained by the Video Pixel Networks (Kalchbrenner et al., 2016) that obtained 87 nats on the test set. Nevertheless, both approaches are not mutually exclusive and instead of a fully connected decoder we could use a similar PixelCNN decoder to generate sprites with higher likelihood. In this first paper we decided instead to focus in defining the statistical framework and interpretable “what” and “where” decoupling.
When running the proposed method in rollout mode, feeding the outputs back as next time step inputs, we were able to generate high likelihood frames for more time steps than with a baseline LSTM. Also, since the sprite to be generated and its position in the frame are decoupled, in rollout mode we can fix the sprite and only use the δxy coming from the network. This way we can generate realistic looking frames for even longer, but after a few frames we observed the digits stopped moving or moved in the wrong direction (see video in the companion code repository). This means that the LSTM RNN was not able to maintain its internal dynamics for too long, thus, there is still room for improvement in the proposed architecture.
In Fig. 5 we show sample rollout videos. The network was fed with 10 frames and asked to generate 10 more getting its own outputs back as inputs and the companion code repository for an animated version of this figure.
This experiment also suggests several improvements in the proposed architecture. For example, we assumed that the internal RNN has to calculate a sprite at every time step, which is inefficient when the sprites don’t change in the video. We should improve the architecture with an extra memory unity that snapshots the sprites and avoid the burden of recalculating the sprites at every step. We believe this would a possible way to free representation power that the internal RNN could use to model the movement dynamics for even more time steps.
5 CONCLUSIONS
This paper introduced a statistical framework for modeling video of 2D scenes inspired by graphics pipelines and variational auto-encoding Bayes. From this statistical framework we derived a variational lower bound that decouples sprites and their dynamics in a video. To optimize this lower bound, we suggested a family of architectures called Perception Updating Networks that can take advantage of this decoupled representation by memorizing sprites or their percepts and updating in location in a scene independently. We showed that this architecture could generate videos that are interpretable and are better suited than baseline RNNs for long video generation.
ACKNOWLEDGMENTS
We thank Ryan Burt for several suggestions to the first draft. This work was partially funded by the University of Florida Graduate Student Fellowship and ONR N00014-14-1-0542.

Review:
This paper presents a generative model of video sequence data where the frames are assumed to be generated by a static background with a 2d sprite composited onto it at each timestep.  The sprite itself is allowed to dynamically change its appearance and location within the image from frame to frame.  This paper follows the VAE (Variational Autoencoder) approach, where a recognition/inference network allows them to recover the latent state at each timestep.
Some results are presented on simple synthetic data (such as a moving rectangle on a black background or the “Moving MNIST” data.  However, the results are preliminary and I suspect that the assumptions used in the paper are far too strong too be useful in real videos.  On the Moving MNIST data, the numerical results are not competitive to state of the art numbers.
The model itself is also not particularly novel and the work currently misses some relevant citations.  The form of the forward model, for example, could be viewed as a variation on the DRAW paper by Gregor et al (ICML 2014).  Efficient Inference in Occlusion-Aware Generative Models of Images by Huang & Murphy (ICLR) is another relevant work, which used a variational auto-encoder with a spatial transformer and an RNN-like sequence model to model the appearance of multiple sprites on a background.
Finally, the exposition in this paper is short on many details and I don’t believe that the paper is reproducible from the text alone.  For example, it is not clear what the form of the recognition model is…  Low-level details (which are very important) are also not presented, such as initialization strategy.