L EARNING TO REPRESENT AND PREDICT IMAGE SEQUENCES VIA POLAR STRAIGHTENING
Anonymous authors
Paper under double-blind review
A BSTRACT
Observer motion and continuous deformations of objects and textures imbue natural videos with distinct temporal structures, enabling partial prediction of future frames from past ones.
Conventional methods ﬁrst estimate local motion, or optic ﬂow, and then use it to predict future frames by warping and copying content.
Here, we explore a more direct methodology, in which frames are mapped into a learned representation space where the structure of temporal evolution is more readily accessible.
Motivated by the geometry of the Fourier shift theorem and its group-theoretic generalization, we formulate a simple architecture that represents video frames in learned polar coordinates to facilitate prediction.
Speciﬁcally, we construct networks in which pairs of convolutional channel coefﬁcients are interpreted as complex-valued, and are expected to evolve with slowly varying amplitudes and linearly advancing phases.
We train these models on next-frame prediction, and compare their performance with that of conventional methods using optic ﬂow, and other learned predictive networks, evaluated on natural videos from two datasets.
We ﬁnd that the polar predictor achieves high prediction performance while remaining interpretable and fast, thereby demonstrating the potential of a ﬂow-free video processing methodology that is trained end-to-end to predict natural video content.
I NTRODUCTION
One way to frame the fundamental problem of vision is that of representing the signal in a form that is more useful for performing visual tasks, be they estimation, recognition, or guiding motor actions.
Perhaps the most general “task” is that of temporal prediction, which has been proposed as a fundamental goal for unsupervised learning of visual representations (F¨oldi´ak, 1991).
But previous research along these lines has generally focused on estimating these transformations rather than using them to predict: extracting slow features (Wiskott & Sejnowski, 2002), and ﬁnding dictionaries and sparse codes that have slow amplitudes and phases (Cadieu & Olshausen, 2012).
In video processing and computer vision, a common strategy for temporal prediction is to ﬁrst estimate local translational motion, and to then copy and paste content to predict the next frame.
Such motion compensation is an important component in making compression schemes like MPEG successful (Wiegand et al., 2003).
These video coding standards are the fruit of major engineering efforts, they make digital video communication feasible and are widely used.
But motion estimation is a difﬁcult nonlinear problem, and existing methods fail in regions where temporal evolution is not translational.
For example, in cases of expanding or rotating motion, discontinuous motion at occlusion boundaries, or mixtures of motion arising from semi-transparent surfaces (e.g., viewing the world through a dirty pane of glass).
In compression schemes, these failures of motion estimation lead to prediction errors, which are then ﬁxed by sending additional corrective bits.
Human perception does not seem to suffer from such failures - at least, our subjective sense is that we can anticipate the time-evolution of visual input even in the vicinity of these commonly occurring non-translational changes.
In fact, those changes are often the most informative ones as they reveal object boundaries, provide ordinal depth and other information about the visual scene.
This may imply that humans make use of a different strategy, perhaps bypassing altogether the estimation of motion, to represent and predict evolving visual input.
Toward this end, and inspired by recent hypotheses that primate visual representations support prediction by “straightening” the temporal trajectories of naturally-occurring input (H´enaff et al., 2019), we formulate an objective for learning an image representation that facilitates prediction by linearizing the temporal trajectories of frames of natural video.
This separation of the instantaneous representation and the temporal prediction is best motivated by considering the behavior of rigidly translating video content when viewed in the frequency domain.
First in section 1.1, we review how translation corresponds to steady phase advancement in the frequency domain, and then in section 1.2, we explain how this relationship reduces prediction of rigidly translating content to angular extrapolation. We place this observation in the general context of group representation theory in section 1.3.
Next in section 2, we describe how to ﬁt parameterized mappings of individual video frames into complex coefﬁcients which can be temporally predicted by phase advancement.
These predicted representations are then used to synthesize an estimated frame, and the entire systems are trained end-to-end to minimize next frame prediction errors.
In section 3, we report training results of several such systems, and show that they produce systematic improvements in predictive performance over conventional motion compensation methods, or direct predictive neural networks. Finally, in section 4, we relate our approach to existing work and then in section 5 discuss its signiﬁcance and implications.
1. B ASE CASE : THE F OURIER SHIFT THEOREM
Our approach is motivated by the well-known behavior of Fourier representations with respect to signal translation.
Speciﬁcally, the complex exponentials that make up the Fourier basis are the eigenfunctions of the translation operator, and translation of inputs produces systematic phase advances of frequency coefﬁcients.
Let x 2 R N be a discrete signal indexed by spatial location n 2 [0 , N  1] , and let e x 2 C N be its Fourier transform indexed by k 2 [0 , N  1] .
We write x v ( n ) =
x ( n  v ) , the translation of x by v modulo N (ie. circular shift with period N).
Deﬁning
=
e i 2 ⇡ /N , the primitive N-th root of unity, we can express the Fourier shift theorem as:
[Equation]
This relationship may be depicted in a compact diagram:
[Equation]
where F indicates the Fourier transform.
In the context of our goals, the diagram illustrates the point that transforming to the Fourier domain renders translation a “simpler” operation: a phase advance is a rotation in the two dimensional (complex) plane.
1. P REDICTION VIA ANGULAR EXTRAPOLATION
Now consider observations of a signal that translates at a constant velocity over time, x ( n, t ) =
y ( n  vt ) .
Although the temporal evolution is easy to describe, the trajectory of the signal is quite complicated, rendering prediction difﬁcult.
As an example, Figure 1 shows a signal consisting of a sum of two sinusoidal components.
Transforming the signal to the Fourier domain simpliﬁes the description.
In particular, the translational motion now corresponds to circular motion of the two (complex-valued) Fourier coefﬁcients associated with the constituent sinusoids.
The motion is further simpliﬁed by a polar coordinate transform to extract phase and amplitude of each Fourier coefﬁcient.
Speciﬁcally, the motion is now along a straight trajectory, with both phases advancing linearly (but at different rates), and both amplitudes constant.
Note that this is a geometric property that holds for any rigidly translating signal, and offers a simple means of predicting content over time.
Indeed, we can use the shift property on x ( n, t +1)
= x v ( n, t ) and observe that prediction [Figure] is now reduced to linear extrapolation of each coefﬁcient’s phase.
We have the three step process:
[Equation]
Since we assume that the motion from time t to t +1 is identical to that from time t  1 to t , the phase advance kv can be computed from the past two representations as kv
[Equation]
where \ z is the phase of the complex number z .
Thus, a polar coordinate transformation in the
Fourier domain converts translational motion into straight trajectories which are predictable via linear phase extrapolation.
1. G ENERALIZATION : REPRESENTING COMMUTATIVE CONTINUOUS GROUPS
Fourier analysis can be seen as a special case of the representation theory of compact commutative Lie groups (Mackey, 1980). In harmonic analysis, the celebrated Peter-Weyl Theorem (1927) guarantees the existence of harmonic basis functions that generalize the Fourier shift property for [Table]
[Figure] tween the predicted and actual frame at t + 1 ). arbitrary compact Lie groups. Speciﬁcally, the linear unitary representation of G is given by an explicit complete orthogonal basis, constructed from the ﬁnite-dimensional irreducible representations of G (Hall, 2013).
The deﬁning property of an irreducible representation is that its span is invariant under group action, so that arbitrary shifts can be expressed as linear combination of basis functions (an example is the construction of steerable ﬁlters (Freeman et al., 1991) in the computational vision literature). In the case of compact commutative Lie groups, complex irreducible representations are one-dimensional, meaning that harmonic basis functions come in pairs.
Therefore, the angular extrapolation mechanism of the previous section can be employed for prediction in a much wider setting than that of translational motion. Since the groups at play in natural videos (and their corresponding harmonic basis functions) are unknown, we aim to learn them from data.  L EARNING TO PREDICT WITH ANGULAR EXTRAPOLATION To generalize beyond translation and the Fourier transform, we aim to learn a representation of video frames that enables prediction via angular extrapolation.
Speciﬁcally, we focus on next frame prediction, and optimize two parameterized mappings: one for the analysis and one for the synthesis transform.
This framework is illustrated in Figure 2, notice its similarity with diagram 1.
2. A RCHITECTURE AND OBJECTIVE FUNCTION
When focusing on a small region in an image sequence, the transformation observed as time passes can be approximated as a local translation.
That is to say, in a spatial neighborhood around position n , we have: x ( n, t + 1)
⇡ x ( n  v, t ) .
We can reuse the setup described for global rigid translation, replacing the Fourier transform with a local convolutional operator (Fleet & Jepson, 1990), processing each spatial neighborhood of the image independently and in parallel, and applying angular extrapolation to the coefﬁcients computed at each position.
Analogous to the Fourier transform, we use the same weights for the encoding and decoding stages, that is to say the analysis operator is the transpose of the synthesis operator.
Sharing these weights reduces the number of parameters and simpliﬁes interpretation of the learned solution.
This “polar predictor” (hereafter PP ) is consistent with the general scheme described in ﬁgure 2 where f w is taken to be linear and convolutional, and g w is its transpose.
In practice, we assumed 64 convolutional channels with ﬁlters of size 17 ⇥ 17 pixels, with no additive constants.
At every position in the image (spatial indices are omitted for clarity of notation), each coefﬁcient y j ( t ) is computed as an inner product between the input x ( t ) and the ﬁlter weights w j of each channel j 2 [0 , 63] : y j ( t ) =
w Tj x ( t ) .
In order to obtain phases, we combine coefﬁcients in pairs, indexed by k 2 [0 , 31] , which can be written as single complex coefﬁcient as: z k ( t ) =
y 2 k
( t )
+ iy 2 k
+1 ( t ) 2 C , and expressed in polar coordinates: z k ( t ) =
a k ( t ) e i ✓ k ( t ) .
This polar coordinate transformation is the only non-linear step used in the polar predictor architecture, it pairs of coefﬁcients and differs from more typical activation functions which operate on each coefﬁcient independently.
With this notation, linear phase extrapolation reduces to ˆ z k ( t + 1)
= a k ( t ) e i ( ✓ k ( t )+  ✓ k ( t )) , where the phase advance  ✓ k ( t ) is equal to the phase difference over the interval from t  1 to t :  ✓ k ( t )
=
✓ k ( t )  ✓ k ( t  1) .
This can be written in a more compact form, using complex arithmetic, as:
[Equation]
where z denotes complex conjugation and | z | the complex modulus.
Note that the intermediate phase variable drops-out when the whole computation is expressed using complex coefﬁcients.
This formulation bundles together phase and amplitude computations and has the beneﬁt of handling phases implicitly, bypassing the phase unwrapping difﬁculty and avoiding the notorious instability of angular variables (indeed phase is unstable when amplitude is low), while still computing the desired angular extrapolation.
In practice, this phase advance can be implemented using real valued elements only, as: [Equation]
where trigonometric identities can be used to calculate cos  ✓ k ( t ) and sin  ✓ k ( t ) without explicitly evaluating phases.
We ﬁnd that such an indirect formulation of phase processing is necessary for the stability of training, as previously noted in the texture modeling literature (Portilla & Simoncelli, 2000).
As a second generalization we use a non-linear representation implemented with deep convolutional neural networks for both for the encoder f w and the decoder g w .
The goal is to learn a transformation in which the coefﬁcients can be predicted using the polar extrapolation mechanism.
This “deep polar predictor” ( deepPP ) model can be written as ˆ x ( t + 1) =
g w ( A ( t ) f
w ( x ( t ))) , where A ( t ) depends on both x ( t ) and x ( t  1) and implements angular extrapolation in the code space as described in equation 3.
This model amounts to a signal factorization into local amplitude and phase in a nonlinear representation. We aim to study the effect of non-linearity separately from that of spatial scale, therefore we chose the number of layers and the kernel sizes to match the effective receptive ﬁeld size of the shallow polar predictor, that is to say units in both models operate on the same size input region in the image.
In practice, both the encoder and the decoder are composed of 4 convolutional layers, each with 64 channels, and using ﬁlter kernels of size 5 ⇥ 5 followed by half-wave rectiﬁcation (ReLU).
In this architecture, the encoder and the decoder are not tied together.
The convolutional kernels w are learned by minimizing the average prediction error, measured as squared error:
[Equation]
We restrict the computation of this prediction error to the center of the image because moving content that enters from outside the video frame is inherently unpredictable.
In practice, we trim a 17-pixel strip from each side. Note that we only perform valid convolutions to avoid artiﬁcial interference with prediction, indeed zero-padding creates undesirable boundary artifacts.
2. C OMPARISON MODELS
We compare our method to the traditional motion compensated coding approach that forms the core of inter-picture coding in well established compression standards such as MPEG.
Block matching is an essential component of these standards, allowing the compression of video content by three orders of magnitude with moderate loss of information.
For each block in a frame, typical coders search for the most similar spatially displaced block in the previous frame (typically measured with MSE), and communicate the displacement coordinates to allow prediction of frame content by translating blocks of the (already transmitted) previous frame.
Speciﬁcally, we implemented a “diamond search” algorithm (Zhu & Ma, 2000) operating on blocks of 8 ⇥ 8 pixels, with a maximal search distance of 8 pixels which balances accuracy of motion estimates and speed of estimation (the search step is computationally intensive). Here, we use the estimated displacements to perform causal motion compensation ( cMC ), using displacement vectors estimated from the last two observed frames ( x t  1 and x t ) to predict the next frame ( x t +1 ) rather than the current one (as in MPEG).
To isolate the effects of the prediction in polar coordinates, we also implemented a predictor that uses linear extrapolation on the responses of a deep neural network ( deepL ), with architecture identical to that of the deep polar predictor.
That is to say, we replace equation 3 by:
ˆ y j ( t +1) = 2 y j ( t )  y j
( t  1) , which amounts to enforcing linear dynamics in the latent space of a non-linear representation.
We also implemented a more direct convolutional neural network predictor ( CNN ), that maps pairs of observed frames to an estimate of the next frame (Mathieu et al., 2016).
Such a mapping jointly transforms and predicts visual signals without explicitly partitioning spatial content representation and temporal feature extrapolation.
Speciﬁcally we used a CNN composed of 20 stages, each consisting of 64 channels, and computed with 3 ⇥ 3 ﬁlters without additive constants, followed by halfwave rectiﬁcation.
Unlike the previous architectures, this model jointly processes pairs of frames.
2. D ATASETS AND TRAINING
To train, test and compare these models, we use the DAVIS dataset (Pont-Tuset et al., 2017), which was originally designed as a benchmark for video object segmentation.
Image sequences in this dataset contain diverse motion of scenes and objects (at different magnitudes and speeds, with ﬁxed and moving camera, etc.), which make next frame prediction challenging.
Each clip is sampled at 25 frames per second, and is approximately 3 seconds long.
The set is subdivided into 60 training videos (4741 frames) and 30 test videos (2591 frames).
We pre-processed the data, converting all frames to monochrome luminance values, and scaling their range to the interval [  1 , 1] .
Frames are reduced by cropping a 256 ⇥ 256 central region, where most of the motion tends to occur, and by down-sampling by a factor of two down to size 128 ⇥ 128 pixels.
We hypothesize the temporal evolution present in natural signals to be sufﬁciently diverse, therefore we do not apply any data augmentation procedures.
We crop videos temporally into brief snippets of 11 frames, which allows for prediction of 9 frames, and process them in batches of size 4.
We train each model for one hundred epochs using the Adam optimizer with default parameters and a learning rate of 3 · 10 (Kingma & Ba, 2015).
The learning rate is halved at epochs 50, 60, 70, 80, 90, 100.
We use batch normalization with no additive terms before every half-wave rectiﬁcation, rescaling by the standard deviation of channel coefﬁcients.
Similarly, we also trained on the larger UCF-101 dataset (Soomro et al., 2012). This dataset, initially designed for action recognition, contains about 2. million frames, which amounts to over 27 hours of video data. Note that the clips are only available in compressed formats and may contain motion artifacts (due to inter-frame coding), unlike the
DAVIS dataset that is available as individual frames.
We used the same pre-processing procedure, except that we only reduced by cropping to a 128 ⇥ 128 central region, without any down-sampling. We employ the same training procedure, except that we only run training for 25 epochs.
We summarize the prediction results in Table 1.First, observe that the predictive algorithms considered in this study perform signiﬁcantly better than the baseline obtained by simply copying the last frame.
Second, the polar predictor (PP) performs about as well as the vanilla convolutional neural 50 PP  0 .0 .0 . PP - c M C test RMSE (a) 0 .0 .0 .
50 PP {+ 0 .0 .0 .0 . PP - C NN test RMSE (b) 0 .0 .0 . 4 deepPP  0 . 0 .0 .
+}
.d
ee p PP - PP test RMSE (c) 0 .0 .0 . 4 deepPP  0 . 0 .0 .d ee p PP - d ee p L test RMSE (d) network (CNN) in terms of test mean squared error on DAVIS.
This is remarkable because the PP model has roughly 30 times fewer parameters and uses a single non-linearity, while the CNN is composed of 20 non-linear layers. This shows that using a polar non-linearity contributes to performance. Finally, observe that deepPP achieves the lowest mean squared error, notably it outperforms the deepL model which is identical in architecture but employs a linear prediction mechanism. It also outperforms the CNN on the UCF-101 test dataset while remaining signiﬁcantly simpler.
This is further evidence that polar coordinates provide an inductive bias beneﬁcial to the prediction task. [Table]
While average performance values provide a compact summary, it is also informative to examine the distribution of prediction errors on individual frames from the test set.
Figure 3 shows pairwise comparison of the predictive algorithms for each frame in the DAVIS dataset.
To make the contrast more apparent, we display performance difference on the vertical axis.
Note that while the models have been optimized to reduce mean squared error, we show root mean squared error (RMSE) in order to facilitate visual inspection of the results (the concavity of the square root spreads out small differences).
We see that (a) The polar predictor representation systematically outperforms causal motion compensation, especially on difﬁcult examples.
(b) The polar predictor outperforms the CNN on the bulk of easy to medium cases but this advantage is reversed for harder examples.
(c)
The deep polar predictor outperforms the single layer polar predictor overall, indicating that the non-linearity in representation can help, but there are many exceptions.
(d) The deep polar predictor clearly outperforms the deep linear predictor, revealing the strong beneﬁt of using a polar extrapolation mechanism over a linear one.
3. L EARNED FILTERS
In order to better understand these results, we visualized the learned ﬁlters of the polar predictor trained on the DAVIS dataset and observed that the learned ﬁlter are Gabor-like (selective for orientation and spatial frequency) and that that they tile the frequency domain.
Interestingly, ﬁlters in a [Figure] [Figure]
pair have a similar frequency amplitude and, in the spatial domain, they are related by a 90 degrees phase shift.
In other words, optimizing for angular extrapolation produces pairs ﬁlters that are in quadrature, see in Figure 9a and 9b in the Appendix.
This relationship is analogous to that of sines and cosines and is consistent with the structure of the angular extrapolation described in equation
[Figure]
3. E XAMPLES
Consider a set of example videos, chosen to illustrate behaviors of the methods being compared.
In Figure 4, we see a wall, its shadow and their sharp boundaries on a grass background as the camera moves.
Both PP and deepPP generate good results, cMC produces a sharp prediction [Figure]
but with signiﬁcant blocking artifacts [Figure] and both the CNN and deepL tend toward excessive blurring.
Figure 5 shows an example with nonrigid motions/deformations.
As the biker advances to the right, the camera tracks and leads its displacement.
Here, the cMC is is signiﬁcantly sharper than the others, but introduces substantial artifacts.
Again, the PP methods produce sharper results than either the CNN or deepL methods.
A couple additional informative examples are displayed in the appendix.
Our method is conceptually related to sparse coding with complex-valued representations (Cadieu & Olshausen, 2012) in that it factorizes natural videos into form and motion. But it differs in a number of important ways: (1) sparse coding focuses on representing, not predicting, the signal; (2) we do not promote sparsity of either amplitude or phase components; (3) ﬁnally, the discontinuity arising from selection of sparse subsets of coefﬁcients seems at odds with the representation of continuous group actions, while our explicit mapping into polar coefﬁcients aims for a smooth and continuous parameterization of the transformations that occur in natural videos.
Several other studies have aimed to learn representations that decompose signal identity and attribute (ie. a what-where , or invariance-equivariance factorization). In particular learning linearized features from video was explored using a heuristic extrapolation mechanism on a toy dataset (Goroshin et al., 2015). The authors developed specialized “soft max-pooling” and “soft argmax-pooling” modules while we rely on principles of signal processing and representation theory to chose the polar non-linearity and describe an effective and stable implementation thereof.
Our method is also related to work that adopts a Lie group formalism in representation learning. Since the seminal work of (Rao & Ruderman, 1998) that proposed learning Lie group generators from dynamic signals, the polar parametrization was explored in (Cohen & Welling, 2014) to identify irreducible representations in a synthetic dataset. The continuous group formalism has also been used in complement to to sparse coding (Chen et al., 2018; Chau et al., 2020) to model natural images as a manifold in the space of pixels. In contrast, our formulation relies on a predictive formulation rather than coding.
Most importantly, our work focuses on local symmetries rather than global signal transformations: a polar predictor is able to adaptively capture and exploit different transformations in different regions of complex high-dimensional signals. Finally, in the ﬂuid mechanics literature, the Koopman operator approach (Mezi´c, 2005) has been used to lift a system from its original state-space to a higher dimensional representation space where its dynamics can be linearized - a dynamical analog of the well known kernel trick. This formalism has spurred a line of work in machine learning that relies on autoencoders to learn coordinate systems that approximately linearize dynamics (Lusch et al., 2018).
In this perspective, our work can also be interpreted as learning the spectral properties of an abstract Koopman operator operating on video data, speciﬁcally estimating its complex eigenvectors.
D ISCUSSION
We have presented a self-supervised representation-learning framework based on temporal prediction of image sequences via polar straightening.
The optimized mappings validate the idea that predictable features can be extracted from natural video using an angular extrapolation in representation space.
Our results also demonstrate that building structure into the architecture can be advantageous, provided that the said structure is well matched to the task.
Speciﬁcally, we rely on the polar coordinate transformation as a bivariate non-linear activation function to pair together coefﬁcients of the representation and show that such an activation function, all else being equal, induces performance beneﬁts.
This exempliﬁes a fundamental theme in computational vision: when possible, let the representation do the analysis.
More concretely, this effect was enabled by bundling angular computations, such that the phase manipulation remains implicit (angular variables are not directly represented in our implementation).
This choice of prediction mechanism, which was motivated by principles of signal processing and harmonic analysis, acts as a structural prior and embodies the general hypothesis that evolving content will continue to evolve in the same way.
Although the conventional deep convolutional network (CNN) considered here could in principle have discovered this solution, it failed to do so (within the constraints of our setup), which emphasizes the importance of architectural design choices that act as inductive biases.
Using terminology from group theory, our polar models factorize signals into an invariant part, which is stable in time, and an equivariant part, which evolves linearly.
This approach to prediction has the advantage of being motion-informed while not relying on explicit motion estimation.
Because it is not constrained to assigning a single motion vector at every location and instead relies on a distribution of phases, this method outperforms the motion compensated algorithm (it also runs several order of magnitude faster).
The polar predictor is not only able to bypass known difﬁculties of motion estimation, but also to generate a representation of each frame.
It would be interesting to validate the generality of that representation by evaluating it on other tasks like object categorization or segmentation, as well as estimation of heading direction of a moving observer.
Several natural extensions of the work presented here can be further explored: (i) treating the angular extrapolation prediction mechanism as a more general building block that is cascaded in a multi-layer architecture; (ii) predicting at longer timescales from representations deeper in the hierarchy; (iii) measuring prediction error directly in the representation domain, while avoiding representation collapse - this would allow a potential connection with biological neural architecture and to human visual perception.
A A PPENDIX
[Table] [Figure] [Figure]
[Figure] [Figure]