Hamiltonian Latent Operators for content and motion disentanglement in image sequences
Anonymous Author(s)
AffiliationAddress email
Abstract
We introduce Halo – a deep generative model utilising HAmiltonian Latent Operators to disentangle content and motion information in image sequences reliably.
The content space captures summary statistics of a sequence, and motion spaceunder a dynamic process determines how information is expressed in any part of thesequence.
By modelling the dynamics as a Hamiltonian motion, important desiderata are ensured: (1) the motion is reversible, (2) the symplectic, volume-preservingstructure in phase space means paths are continuous and are not divergent in thespace.
Consequently, the nearness of sequence frames is realised by the nearness oftheir coordinates in the phase space, which proves valuable for long-term sequencegeneration.
The sequence space is generally composed of different types of dynamical motions.
To ensure long-term separability and perform controlled generation,we associate every motion with a unique Hamiltonian that acts in its respectivesubspace.
We demonstrate the utility of our model by swapping the motion of apair of sequences, controlled generation, and image rotations.
Introduction
The ability to learn to generate artificial image sequences has diverse uses, from animation, key framegeneration, summarisation to restoration and has been explored in previous work over many decades(Hogg, 1983; Hurri and Hyvärinen, 2003; Cremers and Yuille, 2003; Storkey and Williams, 2003;
Kannan et al., 2005).
However, learning to generate arbitrary sequences is not enough; to provideuseful value, the user must be able to control aspects of the sequence generation, such as the motionbeing enacted or the characteristics of the agent doing an action.
To enable this, we must learn todecompose image sequences into content and motion characteristics such that we can apply learntmotions to new objects or vary the types of motions being applied.
Deep generative models (DGMs) such as variational autoencoders (VAEs) (Kingma and Welling,2013) and Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) use neural networks(NNs) to transform the samples from a prior distribution over lower-dimensional latent factors tosamples from the data distribution itself.
Recent developments (Chung et al., 2015; Srivastava et al.
Hsu et al., 2017;
Yingzhen and Mandt, 2018) extend VAEs to sequences using Recurrent Neural
Networks (RNNs) on the representation of temporal frames.
Similar approaches have been taken for
GAN models (Tulyakov et al., 2018; Yoon et al., 2019; Dandi et al., 2020).
The dynamical processes creating the evolution of image sequences are highly constrained.
Considerthe simplistic case of a person walking in a scene with a camera moving around that individual.
Thewalking pose will return to similar positions periodically, and likewise, the revolving camera willrevisit previous positions.
Even without strict periodicity, many dynamical processes are reversible.
Any time that a dynamic could conceivably return to an earlier state suggests an implicit conservationlaw—the conservation of information in the underlying scene generator—as it must be capable of xxxx T q kq kq kq kT p
kp kp kp kT z u k ...
xx t − w x t x T q kt z u k p kt ...
Given any conserved quantity any motion must be modelled in a way that maintains the conservedquantity.
In physics, such a motion is called a Hamiltonian motion; it keeps the corresponding
Hamiltonian function constant.
Hence we argue that a flexible latent Hamiltonian model provides agood inductive bias to learn a representation space that enables conservation of the right quantities(which are themselves learnt) and models the dynamic evolution.
This is mathematically equivalentto saying that we can learn to represent the underlying motions as combinations of differentiablesymmetry groups; all differentiable symmetry transformations follow a conservation law (Noether,1918).
Figure 1 further illustrates the benefit of using Hamiltonian dynamics in the latent spaceof DGM.
1 We observe the dynamics model can discover constant Hamiltonian constant energy inlatent space revealing the relationship among its components proving critical for generating novelenergy preserving sequences.
Therefore, identifying symmetries is a suitable inductive bias fordeveloping expressive DGMs that understand the motion constraints and generalise beyond thetraining data.
Higgins et al.
; Toth et al.
; Botev et al.
(2021) have discussed the benefitsof such inductive biases for learning disentangled representation.
The existing sequential DGMs do not impose any structural prior to constrain the dynamics in motionspace, and therefore, accumulate errors as the length of the sequence grows, quickly deviating fromthe relevant path (Karl et al., 2016;
Fraccaro et al., 2017; Yildiz et al., 2019; Bird and Williams, 2019).
The attractive property of Hamiltonian dynamics is that they are symplectic that is, the divergenceof a vector field is zero, and the evolving dynamics preserves the infinitesimal volume element.
Consequently, the motion paths are restricted to a low-dimensional manifold in the latent space, andthe dynamics can be predicted forward and backwards.
In this paper, we intimate the more general applicability of latent Hamiltonian models; previousapplications have been limited to fairly constrained physical systems.
We propose Halo – a VAEframework to model the dynamics of image sequences using a collection of Hamiltonian operators.
Specifically, for any motion sequence, we model the transition from time step t to step t + 1 using agroup action of a Hamiltonian operator.
The evolution of the dynamics of a sequence leaves certaininformation unchanged, identified as content , and certain properties that evolve in conjunction (i.e.motion ).
Since the space of image sequences can comprise of various types of dynamical actions, wesplit the motion space into subspaces where each subspace models a unique action and is unaffectedby other actions.
This formulation explicitly ensures the separability of dynamics.
It further reducesthe computational cost since the Hamiltonian of the space is now in a block diagonal form whereeach block is a Hamiltonian of a symmetry subgroup.
Here, we focus on a discrete, identified setof actions that we can then compose at generation time.
We want to remark our method can alsowork without action labels as empirically demonstrated in results.
The benefit of identifying actionsapriori is that we can use it for a controlled generation.
We empirically demonstrate the benefits of our approach through i) generation of diverse dynamics from a starting frame and ii) demonstratingsuccessful disentanglement of the content and motion representation.
Related Work
Hamiltonian Neural Networks
In recent years, several deep learning (DL) methods have beenproposed to learn the dynamics of physical systems using Hamiltonian mechanics.
Greydanus et al.(2019) use NNs to predict Hamiltonian from phase-space coordinates s
= ( p , q ) and their derivatives.
In similar work (Bondesan and Lamacraft, 2019) used NNs to discover symmetries of Hamiltonianmechanical systems.
More recently, Hamiltonian NNs are used for simulating complex physicalsystems (Sanchez-Gonzalez et al., 2019, 2020).
The key idea of this work is to represent the states ofparticles as a graph and use a graph neural network (GNN) to predict the change from the currentstate to the next state.
In a follow-up Cranmer et al.
(2020), introduce sparsity on the messages in agraph and use the symbolic regression method to search for physical laws that describe the messagesin the graph.
Recently Toth et al.
(2019) developed the Hamiltonian generative network (HGN),where they proposed to learn a Hamiltonian from image sequences.
HGN maps a sequence to a latentrepresentation and then projects it to the phase space to unroll the dynamics using a symplectic ODEintegrator with Hamilton’s equation.
In another work Yildiz et al.
(2019) use second-order ODEparameterised as a BNN for modelling dynamics of high dimensional sequence data in the latent spaceof VAE.
Most of the developments are built on the neural ODE (Chen et al., 2018) an idea to viewlayers of NNs as internal states of an ODE.
These methods rely on the numerical integration schemeand the stability of the ODE solver.
A Hamiltonian formalism dictates an additional requirement thatthe dynamics of an ODE should be volume-preserving and reversible.
We want to clarify that, unlike
HGN, which mainly focuses on sequence generation and relies on symplectic ODE integrators inthe latent space, we use linear Hamiltonian operators with matrix exponentials and demonstrate itsrelevance for disentanglement.
Latent Space Models There is a long history of latent state space models for modelling sequences (Kalman, 1960; Starner and Pentland, 1997; Roweis and Ghahramani, 1999; Elliott and
Krishnamurthy, 1999; Pavlovic et al., 2000).
More recently, these methods are combined with deepgenerative models for generating high dimensional sequences as well as learning a disentangledrepresentation (Karl et al., 2016; Villegas et al., 2017; Tulyakov et al., 2018; Hsieh et al., 2018;
Yingzhen and Mandt, 2018; Miladinovi ´ c et al., 2019; Minderer et al., 2019; Franceschi et al., 2020;
Zhu et al., 2020).
MoCoGAN (Tulyakov et al., 2018) developed an adversarial framework, combininga random content noise with a sequence of random motion noise to generate videos.
More recently,
DSVAE (Yingzhen and Mandt, 2018) proposed to split a latent space into time-variant and invariantrepresentation and use LSTM (Hochreiter and Schmidhuber, 1997) to model the prior on time-variantrepresentation.
S3VAE (Zhu et al., 2020) improves the disentanglement of DSVAE by minimising amutual information loss between content and motion variables.
Some Hamiltonian methods (Tothet al., 2019; Yildiz et al., 2019) also model the dynamics of high dimensional sequential data in alatent space.
However, the focus in those cases is only on sequence generation, and to our knowledge,this has not been investigated for disentanglement.
Group Transformations in Latent Space Models Rao and Ruderman (1999) proposed the algorithmto model the infinitesimal movement on data manifold using learnable Lie group operators.
Culpepperand Olshausen (2009), use the matrix exponents to learn the transport operators for modelling themanifold trajectory.
Many other similar methods have investigated the use of geometric operators for learning the manifold representation from data (Rao and Ruderman, 1999; Culpepper and
Olshausen, 2009; Memisevic, 2012; Sohl-Dickstein et al., 2010; Cohen and Welling, 2014).
Theuse of symmetries for learning disentangled factor of variations has recently been considered.
Adisentanglement is generally identified as learning representations with independent latent factors.
The main goal is that each latent factor should control a distinct data factor, and a single latentvariable should control no two data factors, Various authors (Bengio et al., 2013; Lake et al., 2017;
Eastwood and Williams, 2018).
Higgins et al.
(2018) have proposed a symmetry-based definition ofdisentanglement.
The goal in these settings was to decompose a latent space into subspaces, and oneach subspace, to learn a unique group transformation such that the subspace is unchanged by theaction of other groups.
Caselles-Dupré et al.
(2019), build such a model using interaction with theenvironment.
Some other similar approaches were recently proposed to learn group transformationsin a latent space (Connor and Rozell, 2020; Quessard et al., 2020; Dupont et al., 2020).
However, the applications were restricted to relatively toy problems and to our knowledge has not been investigatedon higher dimensional videos.
Method
− t t q kt p kt q kt +1 p kt + q kt − 1 p kt −
H k − t t q Kt p Kt q Kt +1 p Kt + q Kt − 1 p Kt −
H K − t t q 1 t p 1 t q 1 t +1 p 1 t + q 1 t − 1 p 1 t −
H...
Encoder
D e c o d e r t ∼U ( { 1 ,...,T } ) momentum position content
Motion Space z
h 1: T h t − w : t ∼ p kt u k q kt q 1: T h t ∼ performing
In the followingwe assume the separation into actionssets are known, but that assumption isrelaxed later.
Let x i 1: T denote the i th image sequence, with x it the t th frame in thesequence.
Let u i be an indicator vector denoting the action associated withthe
i th sequence; i.e. u ik = 1 iff sequence i follows action k and u ik ′ =
0for all other k ′ ̸
= k .
These sequencesand corresponding actions are collected into a dataset { ( x i 1: T , u i ) }
Ni =1 of size N , where, for the sake of simplicity in description, 159 we assume they all are of same length T .
In this paper, we use a latent space to aid modelling of eachsequence and decompose that latent space into two parts, which we call a content space (denoted by
Z ) and a motion space (denoted by S ).
As the data comprises sequences of various actions that takedifferent dynamical form, we further decompose the latent motion space S
= S 1 ⊕ S 2 ⊕ . . .
⊕ S K ,with one part for each action.
In modelling a sequence corresponding to action k , only the subspace
S k will be allowed to change across the length of that sequence.
Each motion subspace is furtherdecomposed into generalised position and momentum parts: S k = ( p k , q k ) .
Only the position part ofthis latent space is used generatively to create an individual image.
The momentum part only affectsthe dynamics.
This formulation model provides many advantages; it prevents the neural networkfrom leaking constant content information via the motion representation, and it ensures the possibilityof preserving key conservation quantities that we argue are implicit in the constraints of most motiondynamics.
This is discussed further in Paragraph 3.
The full framework of our model is illustrated in
Figure 2.
We next introduce the generative model, followed by the variational formalism for inferenceand the learning.
Generative Model For completeness we first present the full probabilistic model in (1-4) beforedescribing each component.
Each dynamic is categorised by a particular action enumerated by k ,encoded in an indicator vector u (i.e. u k = 1 for action k ).
The generative model is conditionedon this action vector.
First, in (1), we sample the content variable z from a prior p ( z ) .
The contentvariable will describe the constant appearance characteristics expressed through the whole sequence.
Next we sample a starting position from prior p ( q k 1 ) , and a momentum from a prior p ( p k 1 ) (weinitialise the actions not represented in the sequence to zero).
The full state-space representation forthe dynamic of action k is then given by s k 1 =
( p k 1 , q k 1 ) .
The dynamical model (3,5) then traces outthe forward trajectory in the phase space.
Finally, we combine the position trajectory with the contentrepresentation and use a decoder neural network to get the emission distribution of the data space sequence.
In summary,
GIVEN: k denoting action label for a sequence,
[Equation]
where d is the dimensionality of k th subspace, m is the dimensionality of data space, f is a dynamicalmodel (5) and ω k are the parameters of f to be used for the k th subspace.
We use an emissiondistribution that is a spherical Gaussian, with a parameterised mean ϕ ( · , · ) , and a [-covariance α 
-]spherical covarianceα 2 I m .
Dynamical Model
In image sequences, we can view each frame of a sequence as a point in anabstract representation space; the temporal dynamics trace a path connecting the frames forming a1-submanifold of the image manifold.
Most dynamical models either try to capture this geometrydeterministically (Srivastava et al., 2015) or probabilistically (Chung et al., 2015; Hsu et al., 2017;
Yingzhen and Mandt, 2018) via linear or non-linear state-space models.
In either case, small errors indynamical steps can accumulate and result in a significant deviation from the manifold when unrollinglong-term trajectories at inference time (Karl et al., 2016; Fraccaro et al., 2017).
Interestingly,
Hamiltonian systems alleviate these issues by constraining the dynamics to be symplectic andreversible.
The symplectic geometry ensures the dynamics are volume-preserving preventing anydeviation from the manifold and reversibility is useful in understanding how the state of an objectchanges under dynamical evolution.
By reversing the arrow of time, the object could return to itsprevious state, this awareness provides a sense of accountability to an object for its actions.
In ourwork, without significant loss of generality, we propose a linear Hamiltonian system in the latentlayer, relying on the deep neural network mapping to data space to handle all nonlinear aspects.
Thelinearity of dynamics also enhances the interpretability of the dynamics.
Definition 1.
A matrix H ∈ R 2 d × 2 d is an Hamiltonian matrix if H T JH =
J , where J is a skewsymmetric matrix J = (cid:18)
I d − I d (cid:19) and I d is an identity matrix.
Consider a coordinate vector s ∈ R 2 d in the phase space S at a time t that evolves under constant
Hamiltonian energy E
= 12 s T M ( t ) s , where M ( t ) is a symmetric matrix.
In Hamiltonian mechanics,the coordinates are specified in terms of position q and momentum p variables as s = ( q , p ) .
Usingthe fact energy E is constant over time we can express equation of motion as, d s ( t ) dt
= JM ( t ) s .
Let
H ( t ) =
JM ( t ) , we can rewrite the equation of motion as, d s ( t ) dt
= H ( t ) s .
The closed-form solution isgiven by matrix exponential s ( t ) =
e t H s (0) .
The matrix exponent has a connection to Lie algebras,and for small t we can interpret e t H as an infinitesimal transformation of s (0) under the action of a
Lie group of H .
We discuss this further in Appendix 2.
For a detailed introduction to the topic werefer to Chevalley (2016).
In this work, we consider K Hamiltonians H 1 , . . .
, H K , each acting on a unique subspace of thephase space S 1 , S 2 , . . .
, S K .
To unroll the trajectory of motion k , we use the group action defined bythe matrix exponent of the operator H k on a starting phase space representation s k 1 ∈ S k given by,
[Equation]
The backward dynamics can simply be obtained by negating time; i.e. replacing t with − t in theabove.
We assume all time steps are equally spaced.
The above formulation provides an explicitdisentanglement of the motion space.
It further allows us to parallelise the computation of matrixexponential by leveraging the block diagonal form of H .
Specific to our work, we parameterisea symmetric matrix M k and obtain its Hamiltonian matrix as H k =
JM k .
The group of suchreal Hamiltonian matrices form a symplectic Lie group under multiplication Sp (2 d ) with 2 d 2 + dindependent elements.
The symplectic geometry proves useful for long term term sequence generation.
We also consider the symplectic orthogonal group SpO (2 d ) that further restricts Hamiltonians toskew-symmetric form with ( d 2 − d ) / 2 independent elements.
The benefit of this restriction is thatthe resulting transforms reduces to rotations which is more interpretable.
We briefly introduce thedetails in Appendix 2.
For a more comprehensive overview, we refer readers to Easton (1993).
Inference We use variational inference to learn the model parameters that leads to maximising theevidence lower bound (ELBO) objective, max q E q ( z , s t | x 1: T , u ) log (cid:104) p ( x 1: T , z , s 1: T | u ) q ( z , s t | x 1: T , u ) (cid:105) , where q ( . | . )
is theapproximate posterior distribution and s t =
[ q t , p t ] .
Since the Hamiltonian dynamics are reversible,at inference time, we randomly sample a choice of frame t and use forward and backward actionof Hamiltonian to trace the trajectory of states after and before that frame for the respective actionas stated in Equation 5.
For a sequence x 1: T , we use the process in (6)) to draw samples from avariational distribution
q ( z , s t | x 1: T , u ) .
Simply, we sample the content variable z conditioned on theobserved data and independently sample the motion states s kt =
[ q kt , p kt ] for the reference frame tconditioned on the observed data and the relevant action k .
Motion states corresponding to otheractions are set to zero.
In equations, this is,
[Equation]
where t is a starting index, q ( q kt | x t , u ) , is the posterior distributions of k th position subspace conditioned on the frame x t and action variable u , q ( p kt | x t − w : t , u ) is the posterior distributions of k thmomentum subspace conditioned on w previous frames and action variable u and q ( z | x 1: T ) is theposterior distribution of the content space conditioned on the entire sequence.
We parameterise thefactorised posterior as a spherical Gaussian distribution learned using an encoder neural network.
Specifically, q ( z | x 1: T ) as a content network, q ( q k t | x t , u ) as a position network, and q ( p k t | x t − w : t , u ) 244 as a momentum network.
We use reparametrisation trick (Kingma and Welling, 2013) to sample fromlatent distribution z = µ + σ ⊙ ϵ where ϵ ∼ N (0 , I ) .
Learning Objective
The learning problem reduces to the optimisation of the following objective,
[Equation]
We have provided the derivation of ELBO in the Appendix 1.
Figure 1, right side, is the probabilisticgraph of the generative and inference model.
Experiments
[Figure]
Rotating Balls We first demonstrate the application of Halo for disentangling content and motion of sequencesof rotating ball that evolves under constraint i 2 + j 2 =
c
where ( i, j ) is a centre of a ball of radius 4 , and c is aconstant. We generated sequences of length 16 under different initial conditions.
We colour half of the sequenceas “red" and the remaining as“blue" to introduce the content element.
We sampled two held out sequences, and in Figure 3 shows the result of swapping their content variable.
This result demonstrates the effectiveness of Haloin disentangling motion from the content .
We next investigate applicability on two complex datasets, i) Sprites ,a sequence of animated characters performing differentactions (‘walking’, ‘spell cast’ and ‘slashing’ from threeviewing angles ‘left’, ‘right’ and ‘straight’) as per spritessheets.
The sequences are of length 8 RGB images ofsize 64 × 64 × 3 .
Each character’s appearance has four 268 attributes: skin colour, hairstyle, tops and pants.
Each attribute can take six values resulting in 1296unique characters.
We used 1000 characters for training and the rest for evaluation.
ii) MUG (Aifantiet al., 2010) a dataset of six facial expressions (anger, disgust, fear, happiness, sadness and surprise)of 52 individuals 3 Sequences are of variable lengths ranging from 50 to 160 frames.
We downsamplethe sequences by a factor of two and use random subsequences of length 8 , crop the face region andresize it to 64 × 64 .
The training and evaluation splits are based on (Tulyakov et al., 2018).
Wealso demonstrate the benefit of our model in predicting rotations on MNIST digits. The symplectic [Table] structure proves useful for sampling long trajectories in the latent space.
The results and discussionon rotating MNIST are in Appendix 3.5.
4.1 Results and Discussion
We first investigate two choices of Hamiltonian structure, a symplectic group using H andthe symplectic orthogonal group using skew- H .
We use fast Taylor approximation (Baderet al., 2019) to compute matrix exponential that provides a stable solution under matrix norms.
[Table]
Quantitative Evaluation We evaluate the generationcapability of the two operators H and skew-
H using the per-frame structural similarity index measure(SSIM) peak signal to noise ratio (PSNR) and meansquared error (MSE).
We generate a more extendedsequence of length 16 (twice the length used fortraining purposes) conditioned on the starting frame.
The sprites consist of periodic sequences of length 8where the start and end frames are identical; in thiscase, we duplicate the sequence to get a ground truthof length 16 .
For MUG, we draw a sequence of length 16 from the evaluation set.
The SSIM scoresare between − 1 and 1 , with a more significant score indicating more similarity between the groundtruth and generated sequence.
Likewise, higher PSNR and lower MSE imply better generation.
Table 1 describes the performance of our model on generation tasks under different scores. Thescores show that our model can generate high-quality sequences from an input image.
We observethat H performs better than skew- H at sequence generation.
The skew- H are interpretable becausethey restrict the Hamiltonian to a rotation matrix.
We hypothesise that the superior performance of Hcan be attributed to the fact that skew- H introduces additional restrictions on the parameters of H thatmight be reducing the expressiveness of the model.
This result is an interesting finding—the questionwe wish to investigate in future work.
For the rest of the paper, we only consider H .
To evaluatedisentanglement we compare with the state-of-the-art baselines DSVAE (Yingzhen and Mandt, 2018),
MoCoGAN(Tulyakov et al., 2018) and S3VAE (Zhu et al., 2020).
We use a pretrained action prediction classifier for evaluating disentanglement.
To begin with, wedraw a starting position and momentum from a prior distribution and use a dynamical model to unrollthe trajectory in the phase space.
Next, we sample the content variable z from real sequences andcombine it with position variables to obtain the image sequence.
We report the performance of theclassifier in predicting the action from these generated sequences.
The score is a useful measure ofmodels tendency to keep the motion intact with the modified content.
We use the same classifierto report the intra-Entropy H ( y | x ) and inter-Entropy H ( y ) that estimates diversity of generatedsequences.
H ( y | x ) measures closeness of generated sequences to the real sequences and H ( y )measures diversity of generated sequences (He et al., 2018).
The results are reported in Table 2.
Halo outperforms the baselines on the MUG and is comparable with S3VAE on sprites.
This improvementresults from explicitly associating an action with a unique subspace that allows separability of thedynamics and avoids any mixing or ambiguity of action in the motion space.
The results on sprites arecomparable; we attribute this to the simplicity of classes in sprites that results in high performanceacross all models.
We want to remark that our formulation is not constrained by action variables u .
Table 2 also describes the results for an unconditional version on the MUG (Aifanti et al., 2010) thatsignificantly outperforms the baseline.
The method details and discussion of the unconditional modelare in the Appendix 3.2.
The benefit of incorporating action variables u is that it allows controlledgeneration of sequences, as demonstrated in Figure 5.
We, next, evaluate the tendency to preserve the identity of sequences.
The identity for sprites isdescribed in terms of four different attributes, and for MUG (Aifanti et al., 2010) it is the sequencelabel of the individual.
We pre-train a classifier on the task of identity prediction and use it forevaluating the generated sequences.
This way, we measure the model’s ability to keep the identityintact when the motion is changed.
For sprites, we report the accuracy of individual attributes.
Table 1outlines the results.
We can see on MUG that our model can preserve the identity with high accuracy.
We can make a similar observation for different attributes of sprites sequences.
Thus, the goodperformance indicates that the content is preserved when traversing the motion subspace, and themotion space is invariant when changing the content variables.
Also validated by the qualitativeresults.
Qualitative Evaluation For the qualitative analysis, we only report results for H as skew- H resultedin similar performance; we omit it here due to limited space. To begin with, the left side of Figure 4gives an example of original, reconstruction and generated sequences.
We generate a sequenceby applying the motion operator on the latent encoding of the first time step.
On the left are theresults for sprites, and on the right of MUG video sequences.
To evaluate disentanglement, we usethe model for the task of motion swapping.
We start by encoding two sequences x 1 1: T and x 2 1: T totheir latent representations ( z 1 , q 11: T ) and ( z 2 , q 21: T ) , next we swap the motion variables ( z 1 , q 21: T )and ( z 2 , q 11: T ) between the two representation spaces, and then pass the resulting representationsthrough the decoder to generate the sequences x 1 → 2 1: T and x 2 → 1 1: T .
The right side of Figure 4 shows theresult.
We can see that swapping the motion part does not affect the identity of the sequences.
Wenow evaluate the image to sequence task to investigate the suitability of our model for a controlledgeneration.
We first encode the image to its content and phase space coordinate in different motionspaces.
Next, use the respective operators to unroll the trajectories in phase space, which are thencombined with content and mapped to the image space using a decoder network.
Figure 5 showsexamples of decoding different motions from the same input image.
For sprites, the actions are inorder ‘walk’, ‘spell card’, ‘slash’ and for MUG they are ordered ‘anger’, ‘disgust’, ‘fear’, ‘happiness’,‘sadness’ and ‘surprise’.
We observe that the visual dynamics associated with all the operators arewell separated.
We also evaluate our model for long-term sequence generation; results are presentedin Appendix 3.2, where it is apparent that longer-term sequences maintain the consistency associatedwith the content-motion pair.
Ablation We carry out ablation to identify the benefits of using Halo over other dynamical methods.
Figure 5 we compare the motion transfer by replacing Hamiltonian with a linear dynamical model and RNN.
We observe that, unlike Halo , the variations in dynamics get constant for Linear and RNN models.
This result shows the benefit of Hamiltonian dynamics as, by definition, they ensure the phase space coordinates change overtime, preventing the encoder neural-network from channelling any static information in themotion space, which is vital for explicit disentanglement of content and motion variables.
[Figure]
More results are provided in Appendix 3.4.
We outline thekey findings in Table 2.
In Figure 4 of the appendix, weshow the caveat with other dynamical models on the controlledgeneration task.
Our formulation proves useful for learningdisentangled representations outperforming various baselines.
We demonstrated its use for the controlled generation of imagesequences.
The effectiveness of our approach is a direct consequence of the symplectic geometry in the phase space, which prevents trajectories from deviatingfrom the motion manifold.
The choice of the quadratic form of energy provides a relationship amonglatent components, which we speculate is critical for the interpretability of actions.
Conclusions and Future Work
We introduced Halo – a DGM that utilises Hamiltonian latent operators to disentangle motion fromthe content in image sequences. Our formulation associates conserved quantities with the dynamics.
Moreover, by the definition of the Hamiltonian dynamics, the motion space has to vary over time; thisprevents the encoder neural network from channelling static information in the motion variables andtherefore provides a helpful notion of disentanglement of content and motion.
Our quantitative resultswith both conditional and unconditional models outperform the existing baselines.
We furthermoredemonstrate disentanglement qualitatively using motion swapping.
In the conditional model, weassociate every action with a unique Hamiltonian that proves critical for the controlled generationtask of an image-to-sequence generation.
Halo can generate long term trajectories and traverse themotion manifolds of different actions in the latent space.
We look forward to future applications toother sequential data types such as molecular trajectories.
A potential limitation of our model is that itis less able to deal with irregularly sampled sequences, changes in tempo or reversals.
In future work,we wish to address this issue by allowing a more flexible prior on the spacing between time steps.
Societal Impact
The applications of generative models for realistic generation of images or videosis an immediate concern for the society.
The proposed application like motion swap and controlledgeneration of videos can be potentially misused for generating fake data.
We restrict the scope of ourpaper to research and make use of data within the condition in the license agreement.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets? Yes ] The data used in the experiments are obtained from public repository.
The code will
be released on publication.
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
Yes
(b) Did you describe the limitations of your work?
[Yes] Yes in section 5(c) Did you discuss any potential negative societal impacts of your work?
[Yes] Yes afterconclusion(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A](b)
Did you include complete proofs of all theoretical results?
[Yes] Provided derivation ofELBO in appendix3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[No] Data is publiclyavailable and the code will be released on publication
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] yes details provided in appendix
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] did that where applicable in Table
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] Yes in the appendix
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes] Provided license of MUG
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[No] There is no offensive content.
The data of face images used in experiments is available from license by an external organisation.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]