U NDERSTANDING IMAGE MOTION WITH GROUP REPRESENTATIONS
{ajaegle, stephi, daphnei, kostas}@seas.upenn.edu
A BSTRACT
Motion is an important signal for agents in dynamic environments, but learning to represent motion from unlabeled video is a difﬁcult and underconstrained problem.
We propose a model of motion based on elementary group properties of transformations and use it to train a representation of image motion.
While most methods of estimating motion are based on pixel-level constraints, we use these group properties to constrain the abstract representation of motion itself.
We demonstrate that a deep neural network trained using this method captures motion in both synthetic 2D sequences and real-world sequences of vehicle motion, without requiring any labels.
Networks trained to respect these constraints implicitly identify the image characteristic of motion in different sequence types.
In the context of vehicle motion, this method extracts information useful for localization, tracking and odometry.
Our results demonstrate that this representation is useful for learning motion in the general setting where explicit labels are difﬁcult to obtain.
I NTRODUCTION
Motion perception is a key component of biological and computer vision.
By understanding how a stream of images reﬂects the motion of the world around it, an agent can better judge how to act.
For example, a ﬂy can use visual motion cues to dodge an approaching hand and to distinguish this threat from a looming landing surface (Reiser & Dickinson (2013)).
Motion is an important cue for understanding actions and predicting 3D scene structure, and it has been extensively studied from computational, ethological, and biological perspectives (Hildreth & Koch (1987)).
In computer vision, the problem of motion representation has typically been approached from either a local or global perspective.
Local representations of motion are exempliﬁed by optical ﬂow.
Flow represents image motion as the 2D displacement of individual pixels of an image, giving rich low-level detail while foregoing a compact representation of the underlying scene motion.
In contrast, global representations such as those used in visual odometry attempt to compactly explain the movement of the whole scene.
Such representations typically rely on a rigid world assumption, thus limiting their applicability to more general settings.
Image transformations due to motion form a subspace of all continuous image transformations.
Smooth changes in the motion subspace correspond to sequences of images with realistic motion.
We wish to characterize this subspace.
The motion subspace differs from other image transformation subspaces, such as changes in the space of images of human faces.
Smooth changes in this space also form a subspace of image transformations, but one containing transformations that do not occur in natural image sequences, such as the face of one person transforming into the face of another.
A representation that characterizes motion should be sensitive to the distinction between image transformations that are realistic (produced by image motion) vs. those that are unrealistic (not produced by image motion).
To be useful for understanding and acting on scene motion, a representation should capture the motion of the observer and all relevant scene content.
Supervised training of such a representation is challenging: explicit motion labels are difﬁcult to obtain, especially for nonrigid scenes where it can be unclear how the structure and motion of the scene should be decomposed.
We propose a framework for learning global, nonrigid motion representations without labels.
While most methods of representing motion rely on pixel-level reconstruction or correspondence to guide learning, our method constrains the representation itself by directly addressing the properties of the latent motion space.
Motion has several properties that we use to operationalize to what extent a model characterizes it.
(1) A model of motion can be read out to estimate metric properties of the motion in the scene, such as the camera translation and rotation.
(2) A model of motion should represent the same motion identically regardless of the image content.
For example, the motion of an object moving to the right should be represented the same whether the object is a cat or a dog.
(3) A model of motion should distinguish sequences produced by natural motion from sequences with image transitions not produced by natural motion, such as cuts.
Here, we present a general model of visual motion and describe how the group properties of visual motion can be used to constrain learning in this model (Figure 1).
We enforce the group properties of associativity and invertibility during training using a metric learning approach (Chopra et al.
on recomposed sequences.
We describe how this technique can be used to train a deep neural network to represent the motion in image sequences of arbitrary length in a low-dimensional, global fashion.
We present evidence that the learned representation captures the global structure of motion in both 2D and 3D settings without labels, hard-coded assumptions about the scene, or explicit feature matching.
[Figure]
properties to learn an approximate group homomorphism Φ ∈ M between motion in the world and in an embedding.
R ELATED W ORK 2. M OTION REPRESENTATIONS
The most common global representations of motion are structure from motion (SfM) and simultaneous localization and mapping (SLAM), which represent motion as a sequence of poses in SE (3) perhaps along with a static point cloud (Scaramuzza & Fraundorfer (2011), Fraundorfer & Scaramuzza (2012)).
These approaches have achieved great success in many applications in recent years, but they are unable to represent non-rigid or independent motions.
The most commonly used local representation is optical ﬂow, which estimates pixel-wise motion over the image, typically constraining it with a smoothness prior (Sun et al.
Scene ﬂow (Wedel et al.
and non-rigid structure from motion (Xiao et al.
(2004)) represent a larger class of 3D motions by generalizing optical ﬂow to the estimation of 3D point trajectories.
These methods represent motion only at local regions (typically single points) and do not attempt to compactly capture the overall motion.
More similar to our approach is work designing or learning spatio-temporal features (STFs) (Laptev (2005)).
STFs are localized and ﬂexible enough to represent non-rigid motions.
They typically include a dimensionality reduction step and hence are somewhat global in purview.
Recent work has used convolutional neural nets (CNNs) to learn task-related STFs directly from images, including
Tran et al.
(2015) and Le (2013).
Unlike our work, both of these approaches are restricted to ﬁxed temporal windows of representation.
Taylor et al.
(2010) uses a standard unsupervised learning technique to learn spatiotemporal features useful for action recognition but not for motion itself.
2. L EARNING REPRESENTATIONS USING VISUAL STRUCTURE
Several recent works have used knowledge of the geometric or spatial structure of images or scenes to train representations.
Doersch et al.
(2015) trains a CNN to classify the correct conﬁguration of image patches to learn the relationship between an image’s patches and its semantic content.
The resulting representation can be ﬁne-tuned for image classiﬁcation.
Yu et al.
(2016) and Patraucean et al.
(2016) train networks to estimate optical ﬂow using the brightness constancy assumption and a smoothness prior as a learning signal.
Zhu et al.
(2017) and Ren et al.
(2017) learn ﬂow using a similar technique.
As with other ﬂow based methods, these works use photometric, local constraints.
Garg et al.
(2016) uses the relationship between depth and disparity to learn to estimate depth from a rectiﬁed stereo camera pair with a known baseline.
Similarly, Konda & Memisevic (2014) treats motion as a latent variable and exploits the relationship between motion and depth to estimate depth.
Other works that learn from sequences typically focus on static image content rather than motion.
Of these, the most similar to ours is Misra et al.
(2016), which shufﬂes the order of images in a sequence to learn representations of image content.
Their approach is designed to capture single image properties that are correlated with temporal order rather than motion itself and their shufﬂing procedure does not preserve the group properties forming the basis of our learning technique.
A related approach is slow feature analysis (Wiskott & Sejnowski (2002)), which is motivated by the notion that slowly varying latents are often behaviorally relevant.
Other works exploring learning from sequences include Jayaraman & Grauman (2015), which learns a representation equivariant to the egomotion of the camera, and Agrawal et al.
(2015), which learns to represent images by explicitly regressing the egomotion between two frames.
Instead of learning to represent motion, these works use labeled
A PPROACH 3. G ROUP PROPERTIES OF MOTION
We base our method on the observation that the set of 3D scene motions, equipped with the composition operation, forms a group.
This group describes the latent structure of transformations in continuous, real-world image sequences.
By learning an embedding that captures the transformations in scenes that occur during motion, we approximate a group homomorphism between the latent motion of the scene and a representation of this motion.
We design our method to capture associativity and invertibility, which allows us to reason about how motions relate and can be composed.
To see that a latent motion space respects these properties ﬁrst consider a latent structure space S .
In this model an element of the structure space generates images I via a projection operator π : S → I .
We also deﬁne a latent motion space M , which is some closed subgroup of the set of homeomorphisms on
S . For any element S of the structure space S , a continuous motion sequence { M t ∈ M | t [-(cid:21) 
 g-]≥ 0 } generates a continuous image sequence { I t [-2 I j t (cid:21) 
-]∈
I | t ≥ 0 } where I t = π ( M t ( S )) .
For a discrete set of images, we can rewrite this as I t = π (( M ∆ t ◦ M t − 1 )( S ))
=
π ( M ∆ t ( S t − 1 )) , which deﬁnes a hidden Markov model, as illustrated in Figure 1 (a). As
M is a closed subgroup of the homeomorphisms on S , it is associative, it contains the identity, and all of its elements have unique inverses in the group.
A simple, speciﬁc case of this model is rigid image motion, such as the motion produced by a camera translating and rotating through a rigid scene in 3D. Here, the latent structure of the scene S can be modeled by a point cloud with a motion space given by M = SE (3) .
For a scene with N rigid bodies, we can describe the motion with a tuple of SE (3) values, M = ( SE (3)) N , where the N th motion acts on the set of points belonging to the N th rigid body.
Generalizing M to general homeomorphisms gives the most general case of motion.
As different scenes contain different degrees of freedom and affordances, it is typically unclear which group effectively characterizes motion in a given real-world setting.
We propose to learn this in a data-driven manner.
3. L EARNING MOTION BY GROUP PROPERTIES
Our goal is to learn a function Φ : I × I → M that maps pairs of images to a representation M of the motion space M .
We also learn a corresponding composition operator (cid:5) :
M → M that emulates the composition of elements in M .
This representation and operator should respect the properties of the motion group in question.
We exploit the structure of the domain to learn to represent and compose motions without labels.
If an image sequence { I t } is sampled from a continuous motion sequence, then the sequence representation should have the following properties for all times t 0 , t 1 , t 2 , t 3 , where t 0 < t 1 < t 2 < t 3 , reﬂecting the group properties of the latent motion:
(i) Associativity: Φ( I t 0 , I t 2 ) (cid:5) Φ( I t 2 , I t 3 ) =
(Φ( I t 0 , I t 1 ) (cid:5) Φ
( I t 1 , I t 2 )) (cid:5) Φ( I t 2 , I t 3 ) = Φ( I t 0 , I t 1 ) (cid:5) (Φ( I t 1 , I t 2 ) (cid:5) Φ( I t 2 , I t 3 )) =
Φ( I t 0 , I t 1 ) (cid:5) Φ( I t 1 , I t 3 ) .
The motion of differently composed subsequences of a sequence are equivalent.
(ii) Existence of the identity element: Φ( I t 0 , I t 1 ) (cid:5) e = Φ( I t 0 , I t 1 )
= e (cid:5) Φ( I t 0 , I t 1 ) , and e = Φ( I t , I t ) for any t .
Null image motion corresponds to the (unique) identity in the latent space.
(iii) Invertibility: Φ( I t 0 , I t 1 ) (cid:5) Φ( I t 1 , I t 0 )
=
e .
The motion of a reversed image sequence is the inverse of the motion of the original image sequence.
We use an embedding loss to approximately enforce associativity and invertibility among subsequences sampled from an image sequence.
Associativity is encouraged by pushing differently composed sequences with equivalent motion to the same representation.
Invertibility of the representation is encouraged by pushing each forward sequence away from its backward counterpart and by pushing all loops to the same representation (i.e. to a learned representation of the identity in the embedding space).
We encourage the uniqueness of the identity representation by pushing loops away from non-identity sequences in the representation.
Because loops have equivalent (identity) motion regardless of scene content, we also push together loops drawn from different sequences.
This procedure is illustrated schematically in Figure 2.
Learning in this framework can be viewed as inference on the graphical model in Figure 1 (a).
Learning a representation of motion is an underconstrained problem, and the group learning rules we introduce here constrain the problem with minimal restriction on the types of scene changes that can be embedded.
In contrast, in optical ﬂow, inference is constrained using the brightness constancy assumption, which assumes that the illumination of a projected scene point does not change between frames (Horn & Schunck (1981)).
Our framework encompasses ﬂow inference if brightness constancy is viewed as a constraint on the projection operator π .
However, the brightness constancy assumption is invalid in many settings.
Our model’s assumptions about geometric properties of motion in the world are valid even over large motions and changing illumination.
The latent structure and motion of a scene are in general non-identiﬁable, which implies that for any given scene, there are several M that can adequately represent M .
We do not claim to learn a unique representation of motion, but rather we attempt to capture one such representation.
Our method assumes the scene has a relatively stable structure, and we do not expect it to handle rapidly changing content or sequence cuts.
We also expect our method to have difﬁculty representing motion in cases of temporally extended occlusion due to the unobservability of motion in such settings.
3. S EQUENCE LEARNING WITH NEURAL NETWORKS
The functions Φ and (cid:5) are implemented as a convolutional neural network (CNN) and a recurrent neural network (RNN), respectively.
We use a long short-term memory (LSTM) RNN (Hochreiter & Schmidhuber (1997)) due to its ability to reliably learn over long time sequences.
The input to the network is in an image sequence [ I 1 , ..., I t ] .
The CNN Φ processes these images and outputs an intermediate representation [ C 1 , 2 , ..., C t − 1 ,t ] .
The LSTM operates over the sequence of CNN outputs to produce an embedding sequence [ R 1 , 2 , ..., R t − 1 ,t ] .
We treat R ( { I t } )
=
R t − 1 ,t as the embedding of sequence { I t } .
This conﬁguration is illustrated schematically in Figure 2 (a).
[Table]
Inequivalent
[Equation]
The network is trained to minimize a hinge loss with respect to the embedding of pairs of sequences:
[Equation]
where d ( R 1 , R 2 ) measures the distance between the embeddings of two example sequences { I 1 t } and { I 2 t (cid:48) } , and m is a ﬁxed scalar margin.
Positive examples are image sequences that are compositionally equivalent, while negative examples are those that are not.
We use the cosine distance for all experiments (with m = 0 . 5 ), as it is smooth and discourages learning the trivial embedding.
In early experiments, results with an L2 distance were similar.
We include six recomposed subsequences for each training sequence: two forward, two backward, and two identity subsequences, as shown in Figure 2 (b).
Subsequences are sampled such that all three sequence types share some of their frames.
To discourage the network from paying attention to only the beginning or end of a sequence, we use several image recomposing schemes.
Forward and backward sequences are sampled to either have the same or different starting frames, and they are drawn from either the same subsequence or from temporally adjacent subsequences.
Because the network is exposed to sequences with the same start and end frames but different motion, this sampling procedure encourages the network to rely on features in the motion domain, rather than on static differences.
During training, we use sequences of varying length to encourage generalization to motions of different temporal scale.
We also explored learning a representation Φ taking single images (and not image pairs) as CNN input.
Because the CNN in this conﬁguration only has access to single images, it cannot extract image motion directly.
In all domains we tested, the representation learned from image pairs outperformed the one learned from single images (Table 1).
E XPERIMENTS
We ﬁrst demonstrate that our learning procedure can discover the structure of motion in the context of rigid scenes undergoing 2D translations and rotations.
We then show that our method learns features useful for representing motion on KITTI (Geiger et al.
(2012)), a dataset of vehicle sequences with motion due to the camera and independent objects.
In all experiments, networks were trained using Adam (Kingma & Ba (2014)).
For MNIST training, we used a ﬁxed decay schedule of 30 epochs and with a starting learning rate chosen by random search (1e-2 was a typical value).
For MNIST, typical batch sizes were 50-60 sequences, and for KITTI (Geiger et al.
the batch sizes were typically 25-30 sequences.
All networks were implemented in Torch (Collobert et al.
[Figure]
We use dilated convolutions to obtain large receptive ﬁelds suitable for capturing large-scale motion patterns.
We used ReLU nonlinearities and batch normalization (Ioffe & Szegedy (2015)) after each convolutional layer.
CNN output was passed to an LSTM with 256 hidden units, followed by a linear layer with 256 hidden units.
In all experiments, CNN-LSTMs were trained on sequences 3-5 images in length.
We tested MNIST networks with sequences of up to 12 images with similar results.
4. R IGID MOTION IN 2D
To test the ability of our learning procedure to represent motion, we trained a network on a dataset consisting of image sequences created from the MNIST dataset.
Each sequence consists of images undergoing a smooth motion drawn from the group of 2D translations and rotations ( SE (2) ) forframes.
We sampled transformation parameters uniformly from [ − 10 , 10] pixels for both horizontal and vertical translation and from [0 , 360) degrees for rotation.
Validation errors are given in Table 1.
We visualize the representation learned on this data using tSNE (van der Maaten & Hinton (2008)) on the sequence embedding for test images undergoing a random translation (Figure 3).
The network representation clearly clusters sequences by both the direction and magnitude of translation.
No obvious clusters appear in terms of the image content.
Similar results were obtained when test data included both translation and rotation.
This suggests that the network has learned a representation that captures the properties of motion in the dataset.
This content-invariant clustering occurs even though the network was never trained to compare images with different spatial content and the same motion.
To further probe the network, we visualize image-conditioned saliency maps in Figure 3.
These saliency maps show the positive (red) and negative (blue) gradients of the network activation with respect to the input image.
As discussed in Simonyan et al.
, such a saliency map can be interpreted as a ﬁrst-order Taylor expansion of the function Φ , evaluated at image I .
The saliency map thus gives an indication of how pixel values affect the representation.
These saliency maps show gradients with respect to the output of the LSTM over the sequence.
[Table]
Intriguingly, these saliency maps bear a strong resemblance to the spatiotemporal energy ﬁlters of classical motion processing (Adelson & Bergen (1985)), which are known to be optimal for 2D speed estimation in natural scenes under certain assumptions Burge & Geisler (2015).
We note that these saliency maps do not simply depict the shape of the ﬁlters of the ﬁrst layers, but rather represent the implicit ﬁlter instantiated by the full network on this image.
When compared across different frames, it becomes clear that the functional mapping learned by the network can ﬂexibly adapt in orientation and arrangement to the image content, unlike standard energy model ﬁlters.
[Table]
R EAL - WORLD MOTION IN 3D
We use the KITTI dataset (Geiger et al.
to test the model’s representation of motion in 3D scenes with camera and independent motion.
We use the representation trained on KITTI tracking in all experiments.
First, we evaluate how well it can decode camera motion.
We compute the representation on all two-frame sequences of KITTI visual odometry, which are labeled with ground truth camera poses.
We then linearly regress from these representations to the change in camera pose between the frames using least squares.
For comparison [Table] we show results using a recent self-supervised ﬂow algorithm Yu et al.
The output of this method is a dense optical ﬂow ﬁeld.
In order to regress from this ﬂow ﬁeld to camera poses, we downsample the ﬂow ﬁelds and run principal component analysis (PCA) over the full training set.
We then linearly regress from the ﬂow ﬁeld PCA components to the camera motion parameters using least squares.
Flow ﬁelds are computed at a resolution of 320x96 pixels, and PCA is computed on downsampled ﬂow ﬁelds of size 160x48 pixels.
We include up to 256 PCA components in the regression.
We refer to this method as Flow+PCA.
Results on held-out test data are displayed in Table 2.
Despite not being trained on any ground truth pose and not seeing any data from the odometry dataset, the learned representation decodes pose consistently better than chance (guessing the mean value).
The largest improvements are in X and Z translation, which also exhibit the most variance in the KITTI odometry dataset.
These results are not competitive with the Flow+PCA results or state-of-the-art odometry methods, but they suggest our method recovers a meaningful representation of motion.
On KITTI visual odometry, our method
[Figure]
Relative Error
[Figure] [Figure]
Relative Error
[Figure]
[Figure]
Relative Error
[Figure]
[Figure]
Relative Error
[Figure]
Relative Error
[Figure]
[Figure]
Relative Error:
[Figure]
Relative Error
[Figure]
Relative Error: 1.
True
Middle Frame 1 F r a m e S t e p s 2 F r a m e S t e p s
[Figure]
Error
[Figure]
[Figure]
Relative Error: 1.
[Figure]
performs similarly to regression to the Flow+PCA method using four to ﬁve principal components 6, which suggests that it is able to capture the dominant global components of motion.
Next, we test the ability of our network to capture the typical motion of the scene by quantifying its performance on an interpolation task.
Given an image sequence [ I 1 , ..., I T ] , we compute the distance between the embedding of the ﬁrst and last frames, R ([ I 1 , I T ]) , vs. the sequence composed of the ﬁrst frame, a middle frame, and the last frame R ([ I 1 , I m , I T ]) (Figure 4).
By comparing the distance when using the true middle frame with the distance when using a different middle frame, we can estimate how sensitive the network is to deviations from the typical dynamics of natural scenes, and hence how well it has learned the relevant motion subgroup.
Results for the full KITTI tracking dataset are shown in Table 3.
We compare our method to a Euclidean distance, computed as the mean pixelwise distance between the probe image and either I 1 or I T (whichever is smaller).
Note that the embedding distance of the true frame is dramatically lower than that of all other frames.
This does not hold for the Euclidean distance, which is often lower for non-interpolating images in the sequence, and is not dramatically different for frames from other sequences.
Finally, we visualize saliency maps on an example sequence in the KITTI dataset in Figure 5.
The saliency map highlights objects moving in the background and the independent motions of the car
[Figure]
in the foreground.
The network highlights areas of the car that can move, such as the bumper, even when these areas don’t contain prominent image differences.
These results suggest our method may be useful for learning features for independent motion detection and tracking.
There are few standard tasks for directly evaluating motion methods beyond odometry.
We attempted to regress from our learned representation to action classes but were unable to obtain competitive performance.
This is not surprising: previous work has shown spatial features are more discriminative for this task, and motion features require extensive processing to be useful (e.g. Simonyan &
Zissermann (2014)).
In future work, we will explore using group properties to encourage intermediate latents to represent motion along with other tasks.
We expect that an embedding that maintains representations of spatial content alongside representations of motion will be more successful in settings like action recognition that depend on both sets of features.
C ONCLUSION
We have presented a new model of motion and a method for learning motion representations.
We have shown that enforcing group properties of motion is sufﬁcient to learn a representation of image motion.
These results suggest that this representation is able to generalize between scenes with disparate content and motion to learn a motion state representation useful for navigation, prediction, and other behavioral tasks relying on motion.
Because of the wide availability of unlabeled video sequences in many settings, we expect our framework to be useful for generating better global motion representations in a variety of real-world tasks.
S UPPLEMENTAL : A DDITIONAL E XPERIMENTS
Here, we expand on the comparison to the self-supervised optical ﬂow baseline given in table 2.
Our method performs equivalently to the Flow+PCA method using the top four to ﬁve principal components (which account for most of the motion variance on KITTI, as shown in Figure 7).
The marginal improvement in Flow+PCA appears to sharply drop off beginning around four to ﬁve principal components as well.
As we saw before, the most dramatic increases in performance come from the Z component of translation and the Y component of rotation, which are the axes of the dominant motion and where chance error is highest.
We note that egomotion estimation beneﬁts greatly from maintaining information about spatial position.
Methods using ﬂow ﬁelds maintain the information by explicitly representing local motion at each position of the image, but our method is global and does not.
KITTI visual odometry is characterized by stereotyped depth and is reasonably modeled as rigid.
Under these circumstances, camera translation and rotation can be estimated from a full ﬂow ﬁeld nearly linearly (Heeger &
Jepson (1992)).
Consistent with this explanation, ﬂow principal components appear to capture both the dominant motions exhibited by the vehicle on this dataset and the stereotyped depth conﬁguration of KITTI (Figure 8).
The good performance of Flow+PCA here highlights the clear advantage of domain-restricted models and learning rules in a setting where those domain restrictions are appropriate.
Our learning rule and model do not make these more restrictive assumptions but still performs reasonably in this setting.