NeMF: Neural Motion Fields for Kinematic Animation
Abstract
We present an implicit neural representation to learn the spatio-temporal space of kinematic motions.
Unlike previous work that represents motion as discrete sequential samples, we propose to express the vast motion space as a continuous function over time, hence the name Neural Motion Fields (NeMF) .
Specifically, we use a neural network to learn this function for miscellaneous sets of motions, which is designed to be a generative model conditioned on a temporal coordinate t and a random vector z for controlling the style.
The model is then trained as a
Variational Autoencoder (VAE) with motion encoders to sample the latent space.
We train our model with a diverse human motion dataset and quadruped dataset to prove its versatility, and finally deploy it as a generic motion prior to solve task-agnostic problems and show its superiority in different motion generation and editing applications, such as motion interpolation, in-betweening, and re-navigating.
More details can be found on our project page: https://cs.yale.edu/homes/ che/projects/nemf/ .
Introduction
Motion synthesis and editing is a core problem in animation and game production, as well as in emerging applications like artificial agents.
Traditional algorithms have limited capability of automatically producing convincing motions with high diversity and complexity.
With the recent availability of large-scale motion capture data, more interest has shifted to deep learning-based methods.
People have adapted various deep learning technologies such as Recurrent Neural Networks [34], Reinforcement Learning [25, 35] and Normalizing Flows [10] for skeletal motion generation.
However, those methods are designed as an auto-regressive process that depends on its own past values and some stochastic terms.
With this constraint, those methods have to sequentially predict the motion at discrete time steps.
Consequently, they cannot just directly infer the motion at certain frames since they must predict the past motion first.
Moreover, with this temporally asymmetric design, it is always hard for those methods to incorporate the control or editing of future frames and, thus, it is difficult to apply them to tasks like motion in-betweening.
Inspired by the recent success in neural radiance fields (NeRF) for novel view synthesis [32], we introduce Neural Motion Fields (NeMF) to model the spatio-temporal space of kinematic motions.
Given that a motion sequence consists of different poses at different time steps, we represent a motion sequence as a continuous function f : t (cid:55)→ f ( t ) which parameterizes the entire sequence by the temporal coordinate t .
This function can be approximated by a multilayer perceptron network (MLP) whose parameters are optimized by minimizing the reconstruction loss between the generated and ground truth motion. Motion Prior
Temporal Field z 1 zf ( t ; z 1 ) tttttf ( t ; z 2 ) t ′t ′t ′t ′t ′
However, the NeMF function defined above overfits one motion sequence.
To extend it to general motion synthesis, as illustrated in Figure 1, we further introduce a random vector z as the conditioning variable to the function as f ( t ; z ) , which now encodes the mapping from the normal distribution to the manifold spanned by all plausible motions.
By varying z , we expect f to output motion with different styles, and by varying t , we obtain the pose at arbitrary time.
We train this generative
NeMF model in the form of a Variational Autoencoder (VAE), where motion sequences are fed to convolutional encoders first to obtain the sampling of z , and then z is passed through an MLP decoder for the reconstruction of the whole motion.
To the best of our knowledge, our paper presents the first continuous and generative implicit motion representation that can sample different types of motion at an arbitrary time step.
This neural motion representation can serve as a generic motion prior to solve task-agnostic problems.
Following Li et al.
[21] who trained a hierarchical human motion prior and applied it in different tasks, we demonstrate utilizing a trained NeMF in various applications, including motion interpolation, motion in-betweening and motion re-navigating.
We formulate those tasks as optimization problems to find the latent variable z that minimize the target energies and restore the complete plausible motion sequences.
Although we did not train this NeMF model for any of these tasks specifically, it achieves performance equivalent or superior to task-specific alternatives in our experiments.
In summary, our contributions are as follows:
• We propose NeMF that represents the continuous motion field as a function over time and design a VAE architecture to train a generative NeMF.
• We validate that our model can reconstruct and synthesize motion with superior quality and diversity than state-of-the-art neural motion priors.
• We demonstrate NeMF in various offline motion editing and creation tasks.
Related Work
To clarify our design choices, we categorize the kinematic motion modeling into two types: time series models and space-time models.
The former approach views motions as the kinematic pose evolving over time, which is often the choice for interactive applications where the future depends on unknown factors such as real-time user inputs.
The latter is our approach, which is suitable for offline applications where we know what to expect for the overall motion within some time range.
2.1 Time Series Models
Time series models are often formulated as an auto-regressive model that predicts the future based on current and past observations.
The predictions are fed into the model again to make further predictions recursively.
GPDM [44] modeled such dynamics of human locomotion over time with Gaussian process.
Recently people studied various deep learning-based auto-regressive architectures, including
Recurrent Neural Networks [5, 29, 49], Reinforcement Learning Networks, Neural ODE [16],
Transformers [1, 22, 36] and other attention models [28].
Many works demonstrated success in real-time interactive kinematic character controls [13, 40–42, 47].
The auto-regressive approach is also suitable for embracing the uncertainty of future predictions with generative models, for example, in the form of VAE as in HuMoR [38] and Motion VAE [25], and in the form of Normalizing Flows as in MoGlow [10].
In applications, Harvey et al.
[9] used LSTM [11] with positional encoding to achieve the state-ofthe-art result for the motion in-betweening task.
But it is limited to a fixed in-betweening setting where some consecutive past frames need to be given as the seed.
One of the key applications of our model is also motion in-betweening, but our method has no constraint on the setting and can generate the missing motion both between motion clips and between sparse keyframes.
2.2 Space-Time Models
Different from time series models, another approach is to directly model the spatio-temporal kinematic state itself.
Such models are often used in combination with a time series model, as in [20] where the dynamic control policy is learned over the GPLVM-based spatial kinematic prior.
One of our inspirations, Motion Fields [19], can be seen as a variant of this approach where the time-series dynamics are learned over the spatial kinematic model represented directly by the data and a handcrafted distance function.
More recently, space-time neural network models overcame the scalability issue in previous methods due to the availability of large-scale datasets [12, 14, 15, 45, 48].
Zhou et al.
[51] demonstrated long-term motion in-betweening by learning the space-time kinematic prior without supervision using Generative Adversarial Networks (GAN) [7].
Kaufmann et al.
[18] also proposed a convolutional network for motion in-betweening.
Our other inspiration, HM-VAE [21], learned a generic motion prior for multiple downstream applications. However, their reliance on a temporally convolutional decoder fixes the output frame rate, thus limiting its application to temporal sub-sampling.
As a space-time model, our work NeMF has the advantage of being able to edit any frames in the motion sequence while maintaining fidelity and the source style compared to the aforementioned works.
2.3 Implicit Neural Representations
Implicit neural representations explosively gained popularity with the success of SIREN [39] and
NeRF [32], which achieved state-of-the-art results in many tasks including novel view synthesis, geometry reconstruction, and solving differential equations.
Among different tasks, the core idea of neural implicit representations is to build a continuous function that maps the spatial or temporal coordinates to any signal at that location while maintaining high-frequency details.
Inspired by their success, our key insight is to interpret time as the parameter of a motion function and learn the landscape of the spatio-temporal kinematics manifold as an implicit motion representation parameterized by temporal coordinates.
Similar ideas have been proposed in modeling time-varying 3D geometries [33] and dynamic scenes [24, 37], and we extend it to the animation domain.
Formulation
In this section, we will first give the definition and notation of our motion representation, then present the formulation of NeMF for a single motion sequence, followed by an extended version of generative
NeMF for the entire motion space.
3.1 Motion Representation
Similar to Zhou et al.
[51] and Li et al.
[21], we divide the motion into two parts: local motion , which contains the pose of the skeleton relative to the root at time t , and global motion , which is the global translation of the root joint.
Following Fussell et al.
[6], we represent the local motion at time t as a matrix X t composed of joint positions x pt ∈ R J × 3 , velocities ˙ x pt ∈ R J × 3 , rotations x rt ∈ R J × 6 in 6D rotation form [50], and angular velocities ˙ x r t ∈ R J × 3 :
[Equation]
where J is the number of joints.
Since all other quantities can be computed from joint rotations, we focus on predicting joint rotations x rt in the formulation.
We then factor out the root orientation r ot ∈ R 6 from local motion by multiplying the inverse of the root transform to each quantity in X t .
In this way, all poses will be transformed to a local space with the same facing direction, thus making them easier to learn for our network.
For the global motion, based on previous works [21, 51], we use a neural network to predict the velocity of the root joint ˙
r t ∈ R 3 as well as its height r ht ∈ R from X t .
We provide details of our global motion predictor in the supplemental.
3.2 Neural Motion Fields
Unlike most other works that represent motion as a discrete sequential process, we represent motion as a continuous vector field of kinematic poses in the temporal domain.
Hence we define this motion field as a function that maps temporal coordinates t to joint rotations and root orientations:
[Equation]
The function f can be approximated by a neural network with the parameters θ that can be optimized by minimizing the reconstruction loss between the generated and ground truth motion, where the ground truth poses can be considered as discrete samples of f at integer time steps.
Similar to NeRF [32], we train an MLP with positional encoding of t to fit a given motion sequence.
Considering a sequence with T frames, we first convert the 6D rotations x rt and r ot to rotation matrices
R t and R ot with the Gram-Schmidt-like process described in [50], and then compute the geodesic distance to measure the rotational difference:
[Equation]
We then evaluate the L 1 loss on local joint positions obtained from forward kinematics (FK) [43], which regularizes the skeletal topology to obtain better results [34]:
[Equation]
The reconstruction loss is finally expressed as a weighted sum of the above terms with weighting factors [Equation] , λ ori , and λ pos :
[Equation]
and we optimize the network parameters θ to minimize this loss function.
3.3 Generative Neural Motion Fields
Although the neural motion field defined above can represent a motion sequence in a compact way, it cannot generate a variety of motions.
To change to a different motion sequence, the entire network needs to be re-trained from scratch, which severely limits its application.
Therefore, in this section, we extend NeMF to a generative model which can represent the entire motion space instead of a specific motion sequence.
First of all, we introduce a conditioning variable z to the input of f , thus parameterizing the entire spatio-temporal kinematics space as a function:
[Equation]
where z defines the spatial location on the manifold and t controls the temporal evolution of the sequence.
We then propose a VAE to formulate Equation 6 as a latent variable model with Gaussian distribution.
Based on our motion representation, the VAE contains two separate convolutional motion encoders to learn and parameterize the posterior distribution of latent variables z l and z g , which control the local motion and root orientation respectively:
[Equation]
where X and r o are the concatenation of all X t and r ot within the same sequence.
The combination of z l and z g forms the final representation of the latent variable z , which are then passed through the MLP decoder to produce joint rotations x rt and root orientations r ot for each time step t , thus, defining the output probability distribution p θ ( x r , r o | z l , z g ) .
In addition to the probabilistic perspective, our model can also be interpreted as learning a motion prior from input sequences, where each latent variable on the prior corresponds to a natural motion sequence.
The MLP decoder then approximates the mapping function between the motion prior and pose space, thus allowing the navigation in the pose space to have a consistent motion style with a fixed z .
During training, we consider the modified variational lower bound:
[Equation]
where the expectation term measures the reconstruction error of the decoder, and the KL divergence
D KL regularizes the encoders’ outputs to be near p ( z l ) and p ( z g ) , which are N ( 0 , I ) in our scenario.
Thus, we formulate the loss function L
=
L rec + λ KL L KL with the weight λ KL to approximate
Equation 8, and the network parameters ( ϕ 1 , ϕ 2 , θ ) are optimized during training to minimize the loss.
3.4 Applications
After training the generative NeMF, we can deploy it as a generic motion prior to solve different tasks via latent space optimization.
Motion In-betweening is a long-standing animation creation problem of generating motion in the interval between two clips or sets of keyframes.
Our insight is, given a sparse set of observations, we can search in the latent space of NeMF to approximate the entire underlying motion thanks to our continuous representation.
Thus, we define the energy function as the reconstruction loss on given frames T , which measures the differences on joint rotations, root orientations, joint positions and root translations:
[Equation]
where L trans weighted by λ trans evaluates the L 1 loss on root joint positions r t predicted from our standalone global motion predictor.
To facilitate convergence, SLERP is first used to perform interpolation in the joint angle space, and then the interpolated inputs are passed through encoders to obtain the initialization of z l and z g .
Motion Re-navigating is a task where we redirect a reference motion with a new trajectory while preserving its style.
More formally, the goal is to generate a motion that 1) looks similar to the given exemplar; and 2) follows the given trajectory as close as possible.
To formulate the energy function, we first borrow ideas from time-series analysis by introducing the soft dynamic time warping metric ( L sDTW ) [4] to measure the similarity between joint positions in the canonical frame [38].
Then, we project the predicted root joints onto the ground plane and compute the L 1 loss between them and the given 2D trajectory.
Last but not least, an additional regularization term is introduced to ensure that the generated and reference motion have a similar angle between their forward direction f t and the tangent direction t t on the trajectory, where the similarity is determined by L sDTW as well.
In total, the energy function we try to minimize can be expressed as:
[Equation]
where λ sim , λ traj and λ angle are three weighting factors introduced to balance different terms.
In our experiments, z l are initialized from the encoder’s output, while z g are optimized from scratch.
Experiments
We train our model on the AMASS dataset [27] for most of the experiments.
After processing, we have roughly 20 hours of human motion sequences at 30 fps for training and testing.
We additionally train our model for the reconstruction experiments on a quadruped motion dataset [47], which contains 30 minutes of dog motion capture at 60 fps.
Details are provided in the supplemental.
4.1 Sanity Test 4.1.1 Motion Reconstruction
We first perform a sanity test to validate our single-motion NeMF model described in Section 3.2, where we test the reconstruction capability of NeMF with different lengths of motions.
For AMASS data, we pick 16 different motion sequences for each length and report the reconstruction errors in Table 1.
For evaluation metrics, we report mean rotation error ( MRE , ◦ ) and mean position error ( MPE , cm ) for each joint, which measure the geodesic distance between joint rotations and
Euclidean distance between root-aligned joint positions.
For the root joint, we further report its orientation error ( MOE , ◦ ), which measures the geodesic distance on the root orientation.
From the results shown in Table 1, we can observe that a simple MLP with positional encoding is able to to achieve very low reconstruction error (lower than 4 mm positional error and 0 .
7 ◦ rotational error) for sequence lengths varying from 32 to 512 frames.
[Table]
We further train our NeMF model on a very long ( 4 , 336 frames and 73 s ) quadruped motion sequence and visualize the result in the supplemental.
Our predicted motion is visually almost identical to the ground truth.
4.1.2 Temporal Sampling
Unlike other motion models, NeMF is theoretically guaranteed to generate smooth motion in arbitrary frame rates.
We find that the dimension of the Fourier features generated by positional encoding plays a critical role in the generalization here. [Figure] [Figure] M e a n P e r - J
o i n t V e l o c i t y [Figure] Since the maximal frequency of the Fourier features is determined by a hyperparameter L as illustrated in [32], we uniformly sample L from 1 to 21 and train NeMF with these different setups.
For each trained model, we respectively generate 30 and 60 fps motions and report their mean per-joint velocity as the smoothness metric in Figure 2.
Although a large value of L does not harm the results with the training frame rate, such a model struggles to generalize to different frame rates.
However, since positional encoding is crucial to achieve satisfactory reconstruction results, we set L = 7 throughout our experiments to balance the trade-off.
In the supplemental video, we show that the generated motion remains smooth even sampled at 240 fps.
4.2 Generative NeMF 4.2.1 Evaluation Metrics
We evaluate the reconstruction capability of our generative NeMF model through direct network inference and motion synthesis capability through latent space sampling.
To better describe motion reconstruction, we further measure the Euclidean distance on the root joint translation ( MTE , cm ) and global joint displacement on the final step ( FDE , cm ) [45].
For estimating the versatile motion quality, we introduce three metrics, namely Fréchet Inception Distance ( FID ), diversity ( Diversity ), and foot skating ( FS ), following related papers [8, 25, 36, 47].
Generally speaking, a lower FID suggests a more natural result, a higher diversity indicates a more various result, and foot skating shows the accumulated drift of foot joints during contact.
Please see our supplemental for details.
4.2.2 Ablation Study
We first take a look at the effect of the positional encoding function.
By removing positional encoding, the input temporal coordinate t will be passed directly to the MLP to infer pose parameters.
Results in the first row of Table 2 show that the model without positional encoding produces higher errors in most metrics we evaluate, thus demonstrating the necessity of positional encoding in terms of the fidelity of the results.
We further experiment on using single motion encoder instead of two separated ones to process both local motion and root orientation.
In this simpler design, the root orientation and local motion are entangled in the same latent space.
From the results we reported in the second row of Table 2, though the simpler version produces a lower reconstruction error on the root orientation, it has worse performance on joint rotations, joint positions and root translations.
[Table]
3 Latent Space Sampling
In our model, we can navigate in the latent space to control the motion styles and synthesize novel motion.
To examine the smoothness of the latent space and see whether our model can blend different styles of motion at the sequence level, we linearly interpolate z from two existing motion sequences and infer the novel ones.
Previous works like [46] demonstrate motion interpolation between similar motion patterns, such as from walking to zombie-style walking.
However, we show in the supplemental video that our model can blend the high-level perceptual style between much harder cases like jumping and punching.
Since we disentangle the latent space for local motion and root orientation, we also experiment to combine z l and z g from different motions.
Through this, we can create interesting editing results like canceling the spinning motion of a pirouette jump (see supplemental).
4.2.4 Comparison with Other Generative Models
We compare our method with other deep learning-based motion priors on motion reconstruction and synthesis in Table 3.
We select HuMoR [38] and HM-VAE [21] to represent the “time series model” and “space-time model” respectively.
[Table]
HuMoR [38]
[-FS ↓ HuMoR [31] 13 .9 .
-][Table]
HM-VAE [-[20] 10 .
-][21] [Table]
Ours 5 [-.2 .
-].
[Table] [Table] Although HuMoR can generate convincing motions, we observe two shortcomings of HuMoR: 1) in motion reconstruction, HuMoR’s results will gradually diverge due to its auto-regressive prediction.
From the quantitative results in Table 3 and qualitative results in Figure 3, divergence can be observed in both the large FDE and global translation difference respectively.
In motion synthesis, HuMoR tends to favor common motions like walking when inferring long sequences, thus yielding a relatively high FID and low diversity.
As for our method, we can get rid of these artifacts since we handle the entire sequence at once instead of predicting the motion frame by frame.
[Figure] Compared to HuMoR and our model, HM-VAE models an over-smoothed latent space, thus filtering out high-frequency details in both the reconstructed and synthesized motions.
In short, our model outperforms these state-of-the-art methods in most of the metrics reported in Table 3, as well as achieving the closest match of the ground truth motion visualized in Figure 3.
4.3 In-betweening Tasks
In this task
[Table] we compare our method with traditional and deep learning-based motion in-betweening methods.
For quantitative evaluation, we use the FID and foot skating metrics only, since reconstruction errors cannot fully reflect the quality of motion for highly-varied plausible solutions.
[Table] [Table]
Motion Clips In-betweening.
In this task, we aim at generating motion in the interval fromframes ( 0 . 33 s ) to 30 frames ( 1 s ) between two clips.
We compare with traditional local interpolation methods including SLERP and Inertialization [2], and deep learning-based methods including Robust
Motion In-betweening (RMI) [9] and HM-VAE [21].
In the results reported in Table 4, our method outperforms all other alternatives quantitatively.
We further experiment on real dancing footage by randomly picking several pairs of videos from AIST++ [23] with their corresponding SMPL [26] parameters. We set the transition length to 30 frames and optimize for a latent variable to produce the gap-filling motion. As demonstrated in Figure 4, our model is capable of producing a natural one-second transition between these real dancing footage.
Sparse Keyframe In-betweening.
In this task, a set of sparse keyframe skeleton poses are given every 5 , 10 , 15 or 20 frames.
We compare our method with SLERP and HM-VAE, and report the quantitative results in Table 5.
Neither RMI nor Inertialization is applicable here since they both require multiple frames at the beginning.
From Table 5, SLERP has the best FID score for interval of 5 frames, but becomes worse with increased interval length.
Compared to SLERP and HM-VAE, ours has much better quantitative and qualitative results, especially for long intervals (Figure 5).
In the onionskin images, though SLERP’s result looks similar to ours, our result has more natural high-frequency details.
HM-VAE fails to reconstruct the poses at keyframes and generates over-smoothed results.
Our result, although different from the ground truth, is visually plausible as in the supplemental video.
[Figure]
4.4 Re-navigating Tasks
In this task, we experiment on redirecting the reference motion to different synthetic trajectories.
Qualitative results are reported in Figure 6.
The original motion style is well preserved in the results, and the character has a natural orientation while walking along the trajectory.
[Figure]
Conclusion
We propose an implicit neural motion representation that defines a continuous motion field over style and time.
We design it to be a generative model for the whole motion space with a learned prior.
We design and test different network architectures and use the trained generative model as a motion prior for solving different tasks like motion interpolation, motion in-betweening and motion re-navigating.
5.1 Limitation and Future Work
In this paper we train the generative NeMF model in the form of a VAE, so it has the limitation of not always giving satisfactory results when sampling the latent space.
A promising direction is to design other generative architectures such as GAN, Normalizing Flows, and Diffusion Models for NeMF.
Besides, though several latent variables may exist that satisfy the optimization constraints, our deterministic optimization process can only find one possible z as the final result, thus limiting probabilistic motion synthesis like MoGlow [10].
In the future, since the optimization problems we formulated are analogous to the latent space optimization problems of StyleGAN [17], how to transfer the algorithm of those successful applications designed for StyleGAN to our NeMF setup is an interesting direction to explore. So far our method focuses on motion modeling and cannot adapt to outputs with different body shapes. On the other hand, some works [3, 30, 31] leverage implicit representations to model skinned articulated objects with given poses. Therefore, it would be interesting to combine our work with them to enable the animation and rendering of a specific character with novel motions.
To keep our model task-agnostic, we choose to solve motion tasks by optimizing z .
But for practical applications, task-specific inference models may be preferred for performance consideration.
To have an end-to-end inference model for real-time applications, a possible solution is to design and train the encoder to directly predict z with different input settings.
5.2 Broader Impact
Our model can be applied for kinematic motion creation in animation and game production.
Due to the limitation of diversity in the training data, our model is not guaranteed to generate feasible but uncommon motions. We cannot guarantee that our model will avoid generating motions that could be perceived as offensive to some viewers. Acknowledgments and Disclosure of Funding
This work was supported in part by NSF Grant No. IIS-2007283.
Many thanks to Ruben Villegas for helpful discussions.
Special thanks to Jiaman Li for sharing with us the pre-trained HM-VAE model.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Section 5.1.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See Section 5.2. (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[No] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See supplemental and the beginning of Section 4.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] We fixed random seeds to 0 throughout our experiments.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See supplemental.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] Our model is trained on the AMASS [27] dataset and a quadruped dataset from [47], we also use AIST++ [23] to test motion in-betweening.
(b) Did you mention the license of the assets?
[Yes] See supplemental.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[No] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?