Hamiltonian latent operators for content and motion disentanglement in image sequences
Anonymous Author(s)
AffiliationAddress email
Abstract
We introduce HALO – a deep generative model utilising HAmiltonian Latent Operators to disentangle content and motion information in image sequences reliably.
The content space captures summary statistics of a sequence, and motion spaceunder a dynamic process determines how information is expressed in any part of thesequence.
By modelling the dynamics as a Hamiltonian motion, important desiderata are ensured: (1) the motion is reversible, (2) the symplectic, volume-preservingstructure in phase space means paths are continuous and are not divergent in thespace.
Consequently, the nearness of sequence frames is realised by the nearness oftheir coordinates in the phase space, which proves valuable for long-term sequencegeneration.
The sequence space is generally composed of different types of dynamical motions.
To ensure long-term separability and perform controlled generation,we associate every motion with a unique Hamiltonian that acts in its respectivesubspace.
We demonstrate the utility of our model by swapping the motion of apair of sequences, controlled generation, and image rotations.
Introduction
The ability to learn to generate artificial image sequences has diverse uses, from animation, key framegeneration, summarisation to restoration and has been explored in previous work over many decades(Hogg, 1983; Hurri and Hyvärinen, 2003; Cremers and Yuille, 2003; Storkey and Williams, 2003;
Kannan et al., 2005).
However, learning to generate arbitrary sequences is not enough; to provideuseful value, the user must be able to control aspects of the sequence generation, such as the motionbeing enacted or the characteristics of the agent doing an action.
To enable this, we must learn todecompose image sequences into content and motion characteristics such that we can apply learntmotions to new objects or vary the types of motions being applied.
Deep generative models (DGMs) such as variational autoencoders (VAEs) (Kingma and Welling,2013) and Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) use neural networks(NNs) to transform the samples from a prior distribution over lower-dimensional latent factors tosamples from the data distribution itself.
Recent developments (Chung et al., 2015; Srivastava et al.
Hsu et al., 2017;
Yingzhen and Mandt, 2018) extend VAEs to sequences using Recurrent Neural
Networks (RNNs) on the representation of temporal frames.
Similar approaches have been taken for
GAN models (Tulyakov et al., 2018; Yoon et al., 2019; Dandi et al., 2020).
The dynamical processes creating the evolution of image sequences are highly constrained.
Consider the simplistic case of a person walking in a scene with a camera moving around that individual.
Thewalking pose will return to similar positions periodically, and likewise, the revolving camera willrevisit previous positions.
Even without strict periodicity, many dynamical processes are reversible.
Any time that a dynamic could conceivably return to an earlier state suggests an implicit conservation law—the conservation of information in the underlying scene generator—as it must be capable of xxxx T q kq kq kq kT p
kp kp kp kT z u k ...
xx t − w x t x T q kt z u k p kt ...
Given any conserved quantity any motion must be modelled in a way that maintains the conservedquantity.
In physics, such a motion is called a Hamiltonian motion; it keeps the corresponding
Hamiltonian function constant.
Hence we argue that a flexible latent Hamiltonian model provides agood inductive bias to learn a representation space that enables conservation of the right quantities(which are themselves learnt) and models the dynamic evolution.
This is mathematically equivalentto saying that we can learn to represent the underlying motions as combinations of differentiablesymmetry groups; all differentiable symmetry transformations follow a conservation law (Noether,1918).
Figure 1 further illustrates the benefit of using Hamiltonian dynamics in the latent spaceof DGM.
1 We observe the dynamics model can discover constant Hamiltonian constant energy inlatent space revealing the relationship among its components proving critical for generating novelenergy preserving sequences.
Therefore, identifying symmetries is a suitable inductive bias fordeveloping expressive DGMs that understand the motion constraints and generalise beyond thetraining data.
Higgins et al.
; Toth et al.
; Botev et al.
(2021) have discussed the benefitsof such inductive biases for learning disentangled representation.
The existing sequential DGMs do not impose any structural prior to constrain the dynamics in motionspace, and therefore, accumulate errors as the length of the sequence grows, quickly deviating fromthe relevant path (Karl et al., 2016;
Fraccaro et al., 2017; Yildiz et al., 2019; Bird and Williams, 2019).
The attractive property of Hamiltonian dynamics is that they are symplectic that is, the divergenceof a vector field is zero, and the evolving dynamics preserves the infinitesimal volume element.
Consequently, the motion paths are restricted to a low-dimensional manifold in the latent space, andthe dynamics can be predicted forward and backwards.
In this paper, we intimate the more general applicability of latent Hamiltonian models; previousapplications have been limited to fairly constrained physical systems.
We propose HALO – a VAEframework to model the dynamics of image sequences using a collection of Hamiltonian operators.
Specifically, for any motion sequence, we model the transition from time step t to step t + 1 using agroup action of a Hamiltonian operator.
The evolution of the dynamics of a sequence leaves certaininformation unchanged, identified as content , and certain properties that evolve in conjunction (i.e.motion ).
Since the space of image sequences can comprise of various types of dynamical actions, wesplit the motion space into subspaces where each subspace models a unique action and is unaffectedby other actions.
This formulation explicitly ensures the separability of dynamics.
It further reducesthe computational cost since the Hamiltonian of the space is now in a block diagonal form whereeach block is a Hamiltonian of a symmetry subgroup.
Here, we focus on a discrete, identified setof actions that we can then compose at generation time.
We want to remark our method can alsowork without action labels as empirically demonstrated in results.
The benefit of identifying actionsapriori is that we can use it for a controlled generation.
We empirically demonstrate the benefits of our approach through i) generation of diverse dynamics from a starting frame and ii) demonstratingsuccessful disentanglement of the content and motion representation.
Related Work
Hamiltonian Neural Networks
In recent years, several deep learning (DL) methods have beenproposed to learn the dynamics of physical systems using Hamiltonian mechanics.
Greydanus et al.(2019) use NNs to predict Hamiltonian from phase-space coordinates s
= ( p , q ) and their derivatives.
In similar work (Bondesan and Lamacraft, 2019) used NNs to discover symmetries of Hamiltonianmechanical systems.
More recently, Hamiltonian NNs are used for simulating complex physicalsystems (Sanchez-Gonzalez et al., 2019, 2020).
The key idea of this work is to represent the states ofparticles as a graph and use a graph neural network (GNN) to predict the change from the currentstate to the next state.
In a follow-up Cranmer et al.
(2020), introduce sparsity on the messages in agraph and use the symbolic regression method to search for physical laws that describe the messagesin the graph.
Recently Toth et al.
(2019) developed the Hamiltonian generative network (HGN),where they proposed to learn a Hamiltonian from image sequences.
HGN maps a sequence to a latentrepresentation and then projects it to the phase space to unroll the dynamics using a symplectic ODEintegrator with Hamilton’s equation.
In another work Yildiz et al.
(2019) use second-order ODEparameterised as a BNN for modelling dynamics of high dimensional sequence data in the latent spaceof VAE.
Most of the developments are built on the neural ODE (Chen et al., 2018) an idea to viewlayers of NNs as internal states of an ODE.
These methods rely on the numerical integration schemeand the stability of the ODE solver.
A Hamiltonian formalism dictates an additional requirement thatthe dynamics of an ODE should be volume-preserving and reversible.
We want to clarify that, unlike
HGN, which mainly focuses on sequence generation and relies on symplectic ODE integrators inthe latent space, we use linear Hamiltonian operators with matrix exponentials and demonstrate itsrelevance for disentanglement.
Latent Space Models There is a long history of latent state space models for modelling sequences (Kalman, 1960; Starner and Pentland, 1997; Roweis and Ghahramani, 1999; Elliott and
Krishnamurthy, 1999; Pavlovic et al., 2000).
More recently, these methods are combined with deepgenerative models for generating high dimensional sequences as well as learning a disentangledrepresentation (Karl et al., 2016; Villegas et al., 2017; Tulyakov et al., 2018; Hsieh et al., 2018;
Yingzhen and Mandt, 2018; Miladinovi ´ c et al., 2019; Minderer et al., 2019; Franceschi et al., 2020;
Zhu et al., 2020).
MoCoGAN (Tulyakov et al., 2018) developed an adversarial framework, combininga random content noise with a sequence of random motion noise to generate videos.
More recently,
DSVAE (Yingzhen and Mandt, 2018) proposed to split a latent space into time-variant and invariantrepresentation and use LSTM (Hochreiter and Schmidhuber, 1997) to model the prior on time-variantrepresentation.
S3VAE (Zhu et al., 2020) improves the disentanglement of DSVAE by minimising amutual information loss between content and motion variables.
Some Hamiltonian methods (Tothet al., 2019; Yildiz et al., 2019) also model the dynamics of high dimensional sequential data in alatent space.
However, the focus in those cases is only on sequence generation, and to our knowledge,this has not been investigated for disentanglement.
Group Transformations in Latent Space Models Rao and Ruderman (1999) proposed the algorithmto model the infinitesimal movement on data manifold using learnable Lie group operators.
Culpepperand Olshausen (2009), use the matrix exponents to learn the transport operators for modelling themanifold trajectory.
Many other similar methods have investigated the use of geometric operators for learning the manifold representation from data (Rao and Ruderman, 1999; Culpepper and
Olshausen, 2009; Memisevic, 2012; Sohl-Dickstein et al., 2010; Cohen and Welling, 2014).
Theuse of symmetries for learning disentangled factor of variations has recently been considered.
Adisentanglement is generally identified as learning representations with independent latent factors.
The main goal is that each latent factor should control a distinct data factor, and a single latentvariable should control no two data factors, Various authors (Bengio et al., 2013; Lake et al., 2017;
Eastwood and Williams, 2018).
Higgins et al.
(2018) have proposed a symmetry-based definition ofdisentanglement.
The goal in these settings was to decompose a latent space into subspaces, and oneach subspace, to learn a unique group transformation such that the subspace is unchanged by theaction of other groups.
Caselles-Dupré et al.
(2019), build such a model using interaction with theenvironment.
Some other similar approaches were recently proposed to learn group transformationsin a latent space (Connor and Rozell, 2020; Quessard et al., 2020; Dupont et al., 2020).
However, the applications were restricted to relatively toy problems and to our knowledge has not been investigatedon higher dimensional videos.
Method
− t t q kt p kt q kt +1 p kt + q kt − 1 p kt −
H k − t t q Kt p Kt q Kt +1 p Kt + q Kt − 1 p Kt −
H K − t t q 1 t p 1 t q 1 t +1 p 1 t + q 1 t − 1 p 1 t −
H...
Encoder
D e c o d e r t ∼U ( { 1 ,...,T } ) momentum position content
Motion Space z
h 1: T h t − w : t ∼ p kt u k q kt q 1: T h t ∼ we HALO 133 of
sequences depicts the with of
of
is simply associated with sequence understood may have very different but dynamic data discussed later) theactions are ‘walking’, cast’ and‘slash’ and the sequences within a setare different individuals performingthe relevant action. In the followingwe assume the separation into actionssets are known, but that assumption isrelaxed later.
Let x i 1: T denote the i th image sequence, with x it the t th frame in thesequence.
Let u i be an indicator vector denoting the action associated withthe
i th sequence; i.e. u ik = 1 iff sequence i follows action k and u ik ′ =
0for all other k ′ ̸
= k .
These sequencesand corresponding actions are collected into a dataset { ( x i 1: T , u i ) }
Ni =1 of size N , where, for the sake of simplicity in description, 159 we assume they all are of same length T .
In this paper, we use a latent space to aid modelling of eachsequence and decompose that latent space into two parts, which we call a content space (denoted by
Z ) and a motion space (denoted by S ).
As the data comprises sequences of various actions that takedifferent dynamical form, we further decompose the latent motion space S
= S 1 ⊕ S 2 ⊕ . . .
⊕ S K ,with one part for each action.
In modelling a sequence corresponding to action k , only the subspace
S k will be allowed to change across the length of that sequence.
Each motion subspace is furtherdecomposed into generalised position and momentum parts: S k = ( p k , q k ) .
Only the position part ofthis latent space is used generatively to create an individual image.
The momentum part only affectsthe dynamics.
This formulation model provides many advantages; it prevents the neural networkfrom leaking constant content information via the motion representation, and it ensures the possibilityof preserving key conservation quantities that we argue are implicit in the constraints of most motiondynamics.
This is discussed further in Paragraph 3.
The full framework of our model is illustrated in
Figure 2.
We next introduce the generative model, followed by the variational formalism for inferenceand the learning.
Generative Model For completeness we first present the full probabilistic model in (1-4) beforedescribing each component.
Each dynamic is categorised by a particular action enumerated by k ,encoded in an indicator vector u (i.e. u k = 1 for action k ).
The generative model is conditionedon this action vector.
First, in (1), we sample the content variable z from a prior p ( z ) .
The contentvariable will describe the constant appearance characteristics expressed through the whole sequence.
Next we sample a starting position from prior p ( q k 1 ) , and a momentum from a prior p ( p k 1 ) (weinitialise the actions not represented in the sequence to zero).
The full state-space representation forthe dynamic of action k is then given by s k 1 =
( p k 1 , q k 1 ) .
The dynamical model (3,5) then traces outthe forward trajectory in the phase space.
Finally, we combine the position trajectory with the contentrepresentation and use a decoder neural network to get the emission distribution of the data space sequence.
In summary,
GIVEN: k denoting action label for a sequence,
[Equation]
where d is the dimensionality of k th subspace, m is the dimensionality of data space, f is a dynamicalmodel (5) and ω k are the parameters of f to be used for the k th subspace.
We use an emissiondistribution that is a spherical Gaussian, with a parameterised mean ϕ ( · , · ) , and a {+covariance α 
+} I m .
Dynamical Model
In image sequences, we can view each frame of a sequence as a point in anabstract representation space; the temporal dynamics trace a path connecting the frames forming a1-submanifold of the image manifold.
Most dynamical models either try to capture this geometrydeterministically (Srivastava et al., 2015) or probabilistically (Chung et al., 2015; Hsu et al., 2017;
Yingzhen and Mandt, 2018) via linear or non-linear state-space models.
In either case, small errors indynamical steps can accumulate and result in a significant deviation from the manifold when unrollinglong-term trajectories at inference time (Karl et al., 2016; Fraccaro et al., 2017).
Interestingly,
Hamiltonian systems alleviate these issues by constraining the dynamics to be symplectic andreversible.
The symplectic geometry ensures the dynamics are volume-preserving preventing anydeviation from the manifold and reversibility is useful in understanding how the state of an objectchanges under dynamical evolution.
By reversing the arrow of time, the object could return to itsprevious state, this awareness provides a sense of accountability to an object for its actions.
In ourwork, without significant loss of generality, we propose a linear Hamiltonian system in the latentlayer, relying on the deep neural network mapping to data space to handle all nonlinear aspects.
Thelinearity of dynamics also enhances the interpretability of the dynamics.
Definition 1.
A matrix H ∈ R 2 d × 2 d is an Hamiltonian matrix if H T JH =
J , where J is a skewsymmetric matrix J = (cid:18)
I d − I d (cid:19) and I d is an identity matrix.
Consider a coordinate vector s ∈ R 2 d in the phase space S at a time t that evolves under constant
Hamiltonian energy E
= 12 s T M ( t ) s , where M ( t ) is a symmetric matrix.
In Hamiltonian mechanics,the coordinates are specified in terms of position q and momentum p variables as s = ( q , p ) .
Usingthe fact energy E is constant over time we can express equation of motion as, d s ( t ) dt
= JM ( t ) s .
Let
H ( t ) =
JM ( t ) , we can rewrite the equation of motion as, d s ( t ) dt
= H ( t ) s .
The closed-form solution isgiven by matrix exponential s ( t ) =
e t H s (0) .
The matrix exponent has a connection to Lie algebras,and for small t we can interpret e t H as an infinitesimal transformation of s (0) under the action of a
Lie group of H .
We discuss this further in Appendix 2.
For a detailed introduction to the topic werefer to Chevalley (2016).
We use fast Taylor approximation (Bader et al., 2019) to compute matrixexponential that provides a stable solution under matrix norms.
In this work, we consider K Hamiltonians H 1 , . . .
, H K , each acting on a unique subspace of thephase space S 1 , S 2 , . . .
, S K .
To unroll the trajectory of motion k , we use the group action defined bythe matrix exponent of the operator H k on a starting phase space representation s k 1 ∈ S k given by,
[Equation]
The backward dynamics can simply be obtained by negating time; i.e. replacing t with − t in theabove.
We assume all time steps are equally spaced.
The above formulation provides an explicitdisentanglement of the motion space.
It further allows us to parallelise the computation of matrixexponential by leveraging the block diagonal form of H .
Specific to our work, we parameterisea symmetric matrix M k and obtain its Hamiltonian matrix as H k =
JM k where J is a fixedskew-symmetric matrix as stated in definition (1).
The group of such real Hamiltonian matricesform a symplectic Lie group under multiplication Sp (2 d ) with 2 d 2 + d independent elements.
The symplectic geometry proves useful for long term sequence generation.
We also consider thesymplectic orthogonal group SpO (2 d ) that further restricts Hamiltonian matrix to a skew-symmetricform with ( d 2 − d ) / 2 independent elements.
The benefit of this restriction is that the resulting transforms reduces to rotations which is more interpretable.
We briefly introduce the details in Appendix 2.
For a more comprehensive overview, we refer readers to Easton (1993).
Inference We use variational inference to learn the model parameters that leads to maximising theevidence lower bound (ELBO) objective, max [Equation] ) (cid:105) , where q ( .
| . ) is theapproximate posterior distribution and s t =
[ q t , p t ] .
Since the Hamiltonian dynamics are reversible,at inference time, we randomly sample a choice of frame t and use forward and backward actionof Hamiltonian to trace the trajectory of states after and before that frame for the respective actionas stated in Equation 5.
For a sequence x 1: T , we use the process in (6) to draw samples from avariational distribution
q ( z , s t | x 1: T , u ) .
Simply, we sample the content variable z conditioned on theobserved data and independently sample the motion states s kt =
[ q kt , p kt ] for the reference frame tconditioned on the observed data and the relevant action k .
Motion states corresponding to otheractions are set to zero.
In equations, this is,
[Equation]
where t is a starting index, q ( q kt | x t , u ) , is the posterior distributions of k th position subspace conditioned on the frame x t and action variable u , q ( p kt | x t − w : t , u ) is the posterior distributions of k thmomentum subspace conditioned on w previous frames and action variable u and q ( z | x 1: T ) is theposterior distribution of the content space conditioned on the entire sequence.
We parameterise thefactorised posterior as a spherical Gaussian distribution learned using an encoder neural network.
Specifically, q ( z | x 1: T ) as a content network, q ( q kt | x t , u ) as a position network, and q ( p kt | x t − w : t , u ) 245 as a momentum network.
We use reparametrisation trick (Kingma and Welling, 2013) to sample fromlatent distribution z = µ + σ ⊙ ϵ where ϵ ∼ N (0 , I ) .
Learning Objective
The learning problem reduces to the optimisation of the following objective,
[Equation]
We have provided the derivation of ELBO in the Appendix 1.
Figure 1, right side, is the probabilisticgraph of the generative and inference model.
Experiments
The implementation details of neural network architecture and training procedure are discussed in
Appendix 3.
We first demonstrate the application of HALO on disentangling content and motion insequences of rotating balls evolving under constant energy.
Next, we investigate the applicability ofour approach on two complex datasets: Sprites and MUG Aifanti et al. Rotating Balls We construct sequences of images of a ball that moves under constraint i 2 + j 2 =
c in orbit underdifferent initial conditions, where ( i, j ) is a centre of a balland c is the distance from the centre of an orbit.
To introduce the content element we colour half of the sequenceas “red" and the remaining as “blue".
In Figure 3 we showthe result of swapping content variable of two held outsequences.
This result demonstrates the effectiveness of HALO in disentangling motion from the content .
Next, we introduce details of the two datasets, followedby a discussion of the results in Section 4.1. Sprites isa sequence of animated characters performing differentactions (‘walking’, ‘spell cast’ and ‘slashing’ from threeviewing angles ‘left’, ‘right’ and ‘straight’) as per spritessheets.
The sequences are of length 8 RGB images ofsize 64 × 64 × 3 .
Each character’s appearance has four 272 attributes: skin colour, hairstyle, tops and pants.
Each attribute can take six values resulting in 1296unique characters.
We used 1000 characters for training and the rest for evaluation.
MUG (Aifantiet al., 2010) a dataset of six facial expressions (anger, disgust, fear, happiness, sadness and surprise) [Table] of 52 individuals. Sequences are of variable lengths ranging from 50 to 160 frames.
We downsamplethe sequences by a factor of two and use random subsequences of length 8 , crop the face region andresize it to 64 × 64 .
The training and evaluation splits are based on (Tulyakov et al., 2018).
Wealso demonstrate the application of our model in predicting rotations on MNIST digits where thesymplectic structure proves useful for sampling long trajectories in the latent space.
We refer to Appendix 3.5 for results.
4.1 Results and Discussion
We first compare two choices of Hamiltonian structure, a symplectic group using H and the symplecticorthogonal group by restricting H to a skew-symmetric form that we refer as skew- H .
Here, we mapa starting frame to the latent space, unroll the trajectory and then map the timesteps to a data spaceusing decoder. We generate sequences of length 16 (twice the length used for training purposes).
[Table]
The sprites consist of periodic sequences of length8 where the start and end frames are identical; inthis case, we duplicate the sequence to get a groundtruth of length 16 .
For MUG, we draw a sequenceof length 16 from the evaluation set.
We comparethe generated sequences with target sequences usingper-frame structural similarity index measure (SSIM),peak signal to noise ratio (PSNR) and mean squarederror (MSE).
The SSIM scores are between − 1 and1 , with a more significant score indicating more similarity between the ground truth and generated sequence.
Likewise, higher PSNR and lower MSEimply better generation.
Table 1 describes the performance under different scores demonstratingour model can generate high-quality sequences from an input image.
We observe that H performsbetter than skew- H .
We hypothesise that the superior performance of H can be attributed to the factthat skew- H introduces additional restrictions on the parameters of H that might be reducing theexpressiveness of the model.
This result is an interesting finding—the question we wish to investigatein future work.
For the rest of the paper, we consider dynamical operator H .
To evaluate disentanglement we compare with the state-of-the-art baselines DSVAE (Yingzhen and
Mandt, 2018), MoCoGAN(Tulyakov et al., 2018) and S3VAE (Zhu et al., 2020).
Quantitative Evaluation We use a pretrained action prediction classifier for evaluating disentanglement.
The architecture is provided in Table 9 of Appendix.
To begin with, we draw a starting positionand momentum from a prior distribution and use a dynamical model to unroll the trajectory in thephase space.
Next, we sample the content variable z from real sequences and combine it with positionvariables to generate image sequences.
We report the performance of the classifier in predicting theaction from these generated sequences.
The score is a useful measure of models tendency to keepthe motion intact with the modified content.
We use the same classifier to report the intra-Entropy H ( y | x ) and inter-Entropy H ( y ) that estimates diversity of generated sequences.
H ( y | x ) measurescloseness of generated sequences to the real sequences and H ( y ) measures diversity of generatedsequences (He et al., 2018).
The results are reported in Table 2.
HALO outperforms the baselineson the MUG and is comparable with S3VAE on sprites.
This improvement results from explicitlyassociating an action with a unique subspace that allows separability of the dynamics and avoids anymixing or ambiguity of action in the motion space.
The results on sprites are comparable; we attributethis to the simplicity of classes in sprites that results in high performance across all models.
We wantto remark that our formulation is not constrained by action variables u .
Table 2 also describes theresults for an unconditional version on the MUG (Aifanti et al., 2010) that significantly outperformsthe baseline.
The details of unconditional model are provided in the Appendix 3.2.
The benefit ofincorporating action variables u is that it allows controlled generation of sequences, as demonstratedin Figure 5.
We, next, evaluate the tendency to preserve the identity of sequences.
For sprites identity refers tofour different attributes, and for MUG (Aifanti et al., 2010) it is the sequence label of the individual.
We pre-train a classifier on the task of identity prediction and use it for evaluating the generatedsequences.
This way, we measure the model’s ability to keep the identity intact when the motion ischanged.
For sprites, we report the accuracy of individual attributes.
Table 1 outlines the results.
Wecan see on MUG that our model can preserve the identity with high accuracy.
We can make a similarobservation for different attributes of sprites sequences.
Thus, the good performance indicates thatthe content is preserved when traversing the motion subspace, and the motion space is invariant whenchanging the content variables.
Also validated by the qualitative results.
Qualitative Evaluation We first evaluate the quality of sequence prediction by comparing the originalsequence, its reconstruction and the generation from an initial frame.
To predict the future timestepsof a sequence, we apply the motion operator on the latent encoding of the first time step. Figure 4on
the left are the results for sprites and on the right of MUG video sequences.
Next, we reportsequences reconstructed by swapping motion variables to evaluate disentanglement.
We start byencoding two sequences x 1 1: T and x 2 1: T to their latent representations ( z 1 , q 1 1: T ) and ( z 2 , q 2 1: T ) , nextwe swap the motion variables ( z 1 , q 21: T ) and ( z 2 , q 11: T ) between the two representation spaces, andthen pass the resulting representations through the decoder to generate the sequences x 1 → 2 1: T and x 2 → 1 1: T .
The right side of Figure 4 shows the result.
We can see that swapping the motion part does notaffect the identity of the sequences.
We now evaluate the image to sequence task to investigate thesuitability of our model for a controlled generation.
We first encode the image to its content andphase space coordinate in different motion spaces.
Next, use the respective operators to unroll thetrajectories in phase space, which are then combined with content and mapped to the image spaceusing a decoder network.
Figure 5 shows examples of decoding different motions from the sameinput image.
For sprites, the actions are in order ‘walk’, ‘spell card’, ‘slash’ and for MUG theyare ordered ‘anger’, ‘disgust’, ‘fear’, ‘happiness’, ‘sadness’ and ‘surprise’.
We observe that thevisual dynamics associated with all the operators are well separated.
We also evaluate our modelfor long-term sequence generation; results are presented in Appendix 3.2, where it is apparent thatlonger-term sequences maintain the consistency associated with the content-motion pair.
Ablation We carry out ablation to identify the benefits of using HALO over other dynamical methods.
Figure 5 we compare the motion transfer by replacing Hamiltonian with alinear dynamical model and RNN.
We observe that, unlike HALO , the variations in dy- namics get constant for Linear and RNN models.
This result shows the benefit of Hamiltonian dynamics as, by definition, they ensure the phase space coordinates change overtime, preventing the encoder neural-network from channelling any static information in themotion space, which is vital for explicit disentanglement of content and motion variables.
[Figure]
The details on positional encoding are provided in Appendix 3.3.
More results are provided in Appendix 3.4.
We outline thekey findings in Table 3.
In Figure 4 of the appendix, weshow the caveat with other dynamical models on the controlledgeneration task.
Our formulation proves useful for learningdisentangled representations outperforming various baselines.
We demonstrated its use for the controlled generation of imagesequences.
The effectiveness of our approach is a direct consequence of the symplectic geometry in the phase space, whichprevents trajectories from deviating from the motion manifold.
The choice of the quadratic form of energy provides a relationship among latent components, whichwe speculate is critical for the interpretability of actions.
Conclusions and Future Work
We introduced HALO – a DGM to disentangle motion from the content in image sequences. Ourformulation utilises Hamiltonian latent operators to associate conserved quantities with the dynamics.
Moreover, by the definition of the Hamiltonian dynamics, the motion space has to vary over time; thisprevents the encoder neural network from channelling static information in the motion variables andtherefore provides a helpful notion of disentanglement of content and motion.
Our quantitative resultswith both conditional and unconditional models outperform the existing baselines.
We furthermoredemonstrate disentanglement qualitatively using motion swapping.
In the conditional model, weassociate every action with a unique Hamiltonian that proves critical for the controlled generationtask of an image-to-sequence generation.
HALO can generate long term trajectories and traverse themotion manifolds of different actions in the latent space.
We look forward to future applications toother sequential data types such as molecular trajectories.
A potential limitation of our model is that itis less able to deal with irregularly sampled sequences, changes in tempo or reversals.
In future work,we wish to address this issue by allowing a more flexible prior on the spacing between time steps.
Societal
Impact The applications of generative models for realistic generation of images or videosis an immediate concern for the society.
The proposed application like motion swap and controlledgeneration of videos can be potentially misused for generating fake data.
We restrict the scope of ourpaper to research and make use of data within the condition in the license agreement.
Checklist
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
Yes
(b) Did you describe the limitations of your work?
[Yes] Yes in section (c) Did you discuss any potential negative societal impacts of your work?
[Yes] Yes after conclusion (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[Yes] Provided derivation ofELBO in appendix3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[No] Data is publiclyavailable and we promise to release the code on publication.(b)
Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] yes all details provided in appendix
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] did that where applicable in Table 1(d)
Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] Yes in the appendix4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
[Table]
(b) Did you mention the license of the assets?
[Yes] Provided license of MUG
[Table] (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]We provide details of our network, implementation additional results in supplementary.
We provide such references in the paper.
[Table]
(d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[N/A]
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identifiableinformation or offensive content?
[Yes] There is no offensive content.
The data of faceimages used in experiments is available from license by an external organisation.
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table] (a) Did you include the full text of instructions given to participants and screenshots, if [Table]
applicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?
[N/A]