H UMAN M OTION D IFFUSION M ODEL
Anonymous authors
Paper under double-blind review
A BSTRACT
Natural and expressive human motion generation is the holy grail of computer animation.
It is a challenging task, due to the diversity of possible motion, human perceptual sensitivity to it, and the difficulty of accurately describing it.
Therefore, current generative solutions are either low-quality or limited in expressiveness.
Diffusion models, which have already shown remarkable generative capabilities in other domains, are promising candidates for human motion due to their many-to-many nature, but they tend to be resource hungry and hard to control.
In this paper, we introduce Motion Diffusion Model (MDM), a carefully adapted classifier-free diffusion-based generative model for the human motion domain.
MDM is transformer-based, combining insights from motion generation literature.
A notable design-choice is the prediction of the sample, rather than the noise, in each diffusion step.
This facilitates the use of established geometric losses on the locations and velocities of the motion, such as the foot contact loss.
As we demonstrate, MDM is a generic approach, enabling different modes of conditioning, and different generation tasks.
We show that our model is trained with lightweight resources and yet achieves state-of-the-art results on leading benchmarks for textto-motion and action-to-motion 1 2 3 .
I NTRODUCTION
Human motion generation is a fundamental task in computer animation, with applications spanning from gaming to robotics.
It is a challenging field, due to several reasons, including the vast span of possible motions, and the difficulty and cost of acquiring high quality data.
For the recently emerging text-to-motion setting, where motion is generated from natural language, another inherent problem is data labeling.
For example, the label ”kick” could refer to a soccer kick, as well as a Karate one.
At the same time, given a specific kick there are many ways to describe it, from how it is performed to the emotions it conveys, constituting a many-to-many problem.
Current approaches have shown success in the field, demonstrating plausible mapping from text to motion (Petrovich et al., 2022;
Tevet et al., 2022; Ahuja & Morency, 2019).
All these approaches, however, still limit the learned distribution since they mainly employ auto-encoders or VAEs (Kingma & Welling, 2013) (implying a one-to-one mapping or a normal latent distribution respectively).
In this aspect, diffusion models are a better candidate for human motion generation, as they are free from assumptions on the target distribution, and are known for expressing well the many-to-many distribution matching problem we have described.
Diffusion models (Sohl-Dickstein et al., 2015; Song & Ermon, 2020; Ho et al., 2020) are a generative approach that is gaining significant attention in the computer vision and graphics community.
When trained for conditioned generation, recent diffusion models (Ramesh et al., 2022; Saharia et al., 2022b) have shown breakthroughs in terms of image quality and semantics.
The competence of these models have also been shown for other domains, including videos (Ho et al., 2022), and 3D point clouds (Luo & Hu, 2021).
The problem with such models, however, is that they are notoriously resource demanding and challenging to control.
In this paper, we introduce Motion Diffusion Model (MDM) — a carefully adapted diffusion based generative model for the human motion domain.
Being diffusion-based, MDM gains from the native aforementioned many-to-many expression of the domain, as evidenced by the resulting motion and
[Figure]
quality and diversity (Figure 1).
In addition, MDM combines insights already well established in the motion generation domain, helping it be significantly more lightweight and controllable.
First, instead of the ubiquitous U-net (Ronneberger et al., 2015) backbone, MDM is transformerbased.
As we demonstrate, our architecture (Figure 2) is lightweight and better fits the temporal and spatially irregular nature of motion data (represented as a collection of joints).
A large volume of motion generation research is devoted to learning using geometric losses (Kocabas et al., 2020; Harvey et al., 2020; Aberman et al., 2020).
Some, for example, regulate the velocity of the motion (Petrovich et al., 2021) to prevent jitter, or specifically consider foot sliding using dedicated terms (Shi et al., 2020).
Consistently with these works, we show that applying geometric losses in the diffusion setting improves generation.
The MDM framework has a generic design enabling different forms of conditioning.
We showcase three tasks: text-to-motion, action-to-motion, and unconditioned generation.
We train the model in a classifier-free manner (Ho & Salimans, 2022), which enables trading-off diversity to fidelity, and sampling both conditionally and unconditionally from the same model.
In the text-to-motion task, our model generates coherent motions (Figure 1) that achieve state-of-the-art results on the HumanML3D (Guo et al., 2022a) and
KIT (Plappert et al., 2016) benchmarks.
Moreover, our user study shows that human evaluators prefer our generated motions over real motions 42% of the time (Figure 4(a)).
In action-to-motion, MDM outperforms the state-of-the-art (Guo et al., 2020; Petrovich et al., 2021), even though they were specifically designed for this task, on the common HumanAct12 (Guo et al., 2020) and
UESTC (Ji et al., 2018) benchmarks.
Lastly, we also demonstrate completion and editing.
By adapting diffusion image-inpainting (Song et al., 2020b; Saharia et al., 2022a), we set a motion prefix and suffix, and use our model to fill in the gap.
Doing so under a textual condition guides MDM to fill the gap with a specific motion that still maintains the semantics of the original input.
By performing inpainting in the joints space rather than temporally, we also demonstrate the semantic editing of specific body parts, without changing the others (Figure 3).
Overall, we introduce Motion Diffusion Model, a motion framework that achieves state-of-the-art quality in several motion generation tasks, while requiring only about three days of training on a single mid-range GPU.
It supports geometric losses, which are non trivial to the diffusion setting, but are crucial to the motion domain, and offers the combination of state-of-the-art generative power with well thought-out domain knowledge.
R ELATED W ORK 2. H UMAN M OTION G ENERATION
Neural motion generation, learned from motion capture data, can be conditioned by any signal that describes the motion.
Many works use parts of the motion itself for guidance.
Some predict motion from its prefix poses (Fragkiadaki et al., 2015; Martinez et al., 2017; Hernandez et al., 2019; Guo et al., 2022b).
Others (Harvey & Pal, 2018; Kaufmann et al., 2020; Harvey et al., 2020; Duan et al., 2021) solve in-betweening and super-resolution tasks using bi-directional GRU (Cho et al., 2014) and Transformer (Vaswani et al., 2017) architectures.
Holden et al.
(2016) use auto-encoder to learn motion latent representation, then utilize it to edit and control motion with spatial constraints such as root trajectory and bone lengths.
Motion can be controlled with a high-level guidance given from action class (Guo et al., 2020;
Petrovich et al., 2021; Cervantes et al., 2022), audio (Li et al., 2021;
Aristidou et al., 2022) and natural language (Ahuja & Morency, 2019; Petrovich et al., 2022).
In most cases authors suggests a dedicated approach to map each conditioning domain into motion.
In recent years, the leading approach for the Text-to-Motion task is to learn a shared latent space for language and motion.
JL2P (Ahuja & Morency, 2019) learns the KIT motion-language dataset (Plappert et al., 2016) with an auto-encoder, limiting one-to-one mapping from text to motion.
TEMOS (Petrovich et al., 2022) and T2M (Guo et al., 2022a) suggest using a VAE (Kingma & Welling, 2013) to map a text prompt into a normal distribution in latent space.
Recently, MotionCLIP (Tevet et al., 2022) leverages the shared text-image latent space learned by CLIP (Radford et al., 2021) to expand text-to-motion out of the data limitations and enabled latent space editing.
The human motion manifold can also be learned without labels, as shown by Holden et al.
V-Poser (Pavlakos et al., 2019), and more recently the dedicated MoDi architecture (Raab et al., 2022).
We show that our model is capable for such an unsupervised setting as well.
2. D IFFUSION G ENERATIVE M ODELS
Diffusion models (Sohl-Dickstein et al., 2015; Song & Ermon, 2020) are a class of neural generative models, based on the stochastic diffusion process as it is modeled in Thermodynamics.
In this setting, a sample from the data distribution is gradually noised by the diffusion process.
Then, a neural model learns the reverse process of gradually denoising the sample.
Sampling the learned data distribution is done by denoising a pure initial noise.
Ho et al.
(2020) and Song et al.
(2020a) further developed the practices for image generation applications.
For conditioned generation, Dhariwal &
Nichol (2021), introduced classifier-guided diffusion, which was later on adapted by GLIDE (Nichol et al., 2021) to enable conditioning over CLIP textual representations.
The Classifier-Free Guidance approach Ho & Salimans (2022) enables conditioning while trading-off fidelity and diversity, and achieves better results (Nichol et al., 2021).
In this paper, we implement text-to-motion by conditioning on CLIP in a classifier-free manner, similarly to text-to-image (Ramesh et al., 2022; Saharia et al., 2022b).
Local editing of images is typically defined as an inpainting problem, where a part of the image is constant, and the inpainted part is denoised by the model, possibly under some condition (Song et al., 2020b; Saharia et al., 2022a).
We adapt this technique to edit motion’s specific body parts or temporal intervals (in-betweening) according to an optional condition. Closer to our context, Gu et al.
(2022) used the diffusion formulation to model the stochasticity of human trajectory prediction.
More recently, concurrent to this work, Zhang et al.
and Kim et al.
(2022) have suggested diffusion models for motion generation.
Our work requires significantly fewer GPU resources and makes design choices that enable geometric losses, which improve results.
M OTION D IFFUSION M ODEL
An overview of our method is described in Figure 2.
Our goal is to synthesize a human motion x 1: N of length N given an arbitrary condition c .
This condition can be any real-world signal that will dictate the synthesis, such as audio (Li et al., 2021;
Aristidou et al., 2022), natural language (text-to-motion) (Tevet et al., 2022;
Guo et al., 2022a) or a discrete class (action-to-motion) (Guo et al., 2020;
Petrovich et al., 2021).
In addition, unconditioned motion generation is also possible, which we denote as the null condition c
= ∅ .
The generated motion x 1: N = { x i } N i =1 is a sequences
CLIP text
PE
Linear 𝑥 !"
𝑥 !
# 𝑥 !
$ 𝑥 !
% 𝑧 !& #𝑥 ’"
#𝑥 ’# #𝑥 ’$ #𝑥 ’%
Linear
Transformer Encoder
Linear
Text prompt
MLP 𝑡 + . . .
𝑐 + + + + +
MDM 𝓝(0,𝐼)
Diffuse 𝟎→(𝑻−𝟏) 𝑥 ()" 𝑥 ’ 𝑥 " 𝑥 ( #𝑥 ’ 𝑇 𝑐
MDM
Diffuse 𝟎→(𝑻−𝟐) 𝑥 ()# #𝑥 ’ (𝑇 −1)𝑐
MDM 1 𝑐
Random Mask
Framework.
Diffusion is modeled as a Markov noising process, { x 1: N t } Tt =0 , where x 1: Nis drawn from the data distribution and
[Equation]
where α t ∈ (0 , 1) are constant hyper-parameters.
When α t is small enough, we can approximate x 1: N T ∼ N (0 , I ) .
From here on we use x t to denote the full sequence at noising step t .
In our context, conditioned motion synthesis models the distribution p ( x 0 | c ) as the reversed diffusion process of gradually cleaning x T .
Instead of predicting ϵ t as formulated by Ho et al.
, we follow Ramesh et al.
and use an equivalent formulation to predict the signal itself, i.e., ˆ x 0 = G ( x t , t, c ) with the simple objective (Ho et al., 2020),
[Equation]
Geometric losses.
In the motion domain, generative networks are standardly regularized using geometric losses Petrovich et al.
; Shi et al.
These losses enforce physical properties and prevent artifacts, encouraging natural and coherent motion.
In this work we experiment with three common geometric losses that regulate (1) positions (in case we predict rotations), (2) foot contact, and (3) velocities.
[Equation]
In case we predict joint rotations, FK ( · ) denotes the forward kinematic function converting joint rotations into joint positions (otherwise, it denotes the identity function).
f i ∈ { 0 , 1 } J is the binary foot contact mask for each frame i .
Relevant only to feet, it indicates whether they touch the ground, and are set according to binary ground truth data (Shi et al., 2020).
In essence, it mitigates the foot-sliding effect by nullifying velocities when touching the ground.
Overall, our training loss is
[Equation]
Model.
Our model is illustrated in Figure 2.
We implement G with a straightforward transformer (Vaswani et al., 2017) encoder-only architecture.
The transformer architecture is temporally aware, enabling learning and generating variable-length motions, and is well-proven for the motion domain (Petrovich et al., 2021;
Duan et al., 2021; Aksan et al., 2021).
The noise time-step t and the condition code c are each projected to the transformer dimension by separate feed-forward networks, then summed to yield the token z tk .
Each frame of the noised input x t is linearly projected into the transformer dimension and summed with a standard positional embedding.
z tk and the projected frames are then fed to the encoder.
Excluding the first output token (corresponding to z tk ), the encoder result is projected back to the original motion dimensions, and serves as the prediction ˆ x 0 .
We implement text-to-motion by encoding the text prompt to c with CLIP (Radford et al., 2021) text encoder, and action-to-motion with learned embeddings per class.
Sampling from p ( x 0 | c ) is done in an iterative manner, according to Ho et al.
In every time step t we predict the clean sample ˆ x 0 = G ( x t , t, c ) and noise it back to x t − 1 .
This is repeated from t =
T
until x 0 is achieved (Figure 2 right).
We train our model G using classifier-free guidance (Ho & Salimans, 2022).
In practice, G learns both the conditioned and the unconditioned distributions by randomly setting c = ∅ for 10% of the samples, such that G ( x t , t, ∅ ) approximates p ( x 0 ) .
Then, when sampling G we can trade-off diversity and fidelity by interpolating or even extrapolating the two variants using s :
[Equation]
Editing.
We enable motion in-betweening in the temporal domain, and body part editing in the spatial domain, by adapting diffusion inpainting to motion data.
Editing is done only during sampling, without any training involved.
Given a subset of the motion sequence inputs, when sampling the model (Figure 2 right), at each iteration we overwrite ˆ x 0 with the input part of the motion.
This encourages the generation to remain coherent to original input, while completing the missing parts.
In the temporal setting, the prefix and suffix frames of the motion sequence are the input, and we solve a motion in-betweening problem (Harvey et al., 2020).
Editing can be done either conditionally or unconditionally (by setting c = ∅ ).
In the spatial setting, we show that body parts can be re-synthesized according to a condition c while keeping the rest intact, through the use of the same completion technique.
E XPERIMENTS
We implement MDM for three motion generation tasks: Text-to-Motion(4.1), Action-to-Motion(4.2) and unconditioned generation(5.2.
Each sub-section reviews the data and metrics of the used benchmarks, provides implementation details, and presents qualitative and quantitative results.
Then, we show implementations of motion in-betweening (both conditioned and unconditioned) and bodypart editing by adapting diffusion inpainting to motion (5.1).
Our models have been trained with
T = 1000 noising steps and a cosine noise schedule.
In Appendix A.1, we experiment with different values of T .
All of them have been trained on a single NVIDIA GeForce RTX 2080 Ti GPU for a period of about 3 days.
4. T EXT - TO -M OTION
Text-to-motion is the task of generating motion given an input text prompt.
The output motion is expected to be both implementing the textual description, and a valid sample from the data distribution (i.e. adhering to general human abilities and the rules of physics).
In addition, for each text prompt, we also expect a distribution of motions matching it, rather than just a single result.
We evaluate our model using two leading benchmarks - KIT (Plappert et al., 2016) and HumanML3D (Guo et al., 2022a), over the set of metrics suggested by Guo et al.
(2022a): R-precision and Multimodal-Dist measure the relevancy of the generated motions to the input prompts, FID measures the dissimilarity between the generated and ground truth distributions (in latent space), Diversity measures the variability in the resulting motion distribution, and MultiModality is the average variance given a single text prompt.
For the full implementation of the metrics, please refer to Guo et al.
(2022a).
We use HumanML3D as a platform to compare different backbones of our model, discovering that
[Figure]
the diffusion framework is relatively agnostic to this attribute.
In addition, we conduct a user study comparing our model to current art and ground truth motions.
Data.
HumanML3D is a recent dataset, textually re-annotating motion capture from the
AMASS (Mahmood et al., 2019) and HumanAct12 (Guo et al., 2020) collections.
It contains 14 ,motions annotated by 44 , 970 textual descriptions.
In addition, it suggests a redundant data representation including a concatenation of root velocity, joint positions, joint velocities, joint rotations and the foot contact binary labels.
We also use in this section the same representation for the KIT dataset, brought by the same publishers.
Although limited in the number ( 3 , 911 ) and the diversity of samples, most of the text-to-motion research is based on KIT, hence we view it as important to evaluate using it as well.
Implementation.
In addition to our Transformer encoder-only backbone (Section 3), we experiment
MDM with three more backbones: (1) Transformer decoder injects z tk through the cross-attention layer, instead of as an input token.
(2) Transformer decoder + input token , where z tk is injected both ways, (3) GRU (Cho et al., 2014) concatenate z tk to each input frame , and (4) a U-net adaptation for motion data (Table 1).
U-net is adapted to motion by replacing the 2D convolution filters with 1D convolutions in the temporal axis, such that joints are considered as channels.
This is due to the non-continuous behavior of the joints axis.
Irregular again? Our models were trained with batch size 64 , 8 layers (except GRU that was optimal at 2 ), and latent dimension 512 .
To encode the text we use a frozen CLIP-ViT-B/32 model.
In addition, we experiment with replacing CLIP with sentence-BERT (Reimers & Gurevych, 2019) - a BERT-based (Devlin et al., 2019) text encoder.
The full details will be found in our code to be published and Appendix C. Each model was trained for 500 K steps, afterwhich a checkpoint was chosen that minimizes the FID metric to be reported.
Since foot contact and joint locations are explicitly represented in HumanML3D, we don’t apply geometric losses in this section.
We evaluate our models with guidance-scale s = 2 . 5 which provides a diversity-fidelity sweet spot (Figure 4).
Quantitative evaluation.
We evaluate and compare our models to current art (JL2P Ahuja &
Morency (2019), Text2Gesture (Bhattacharya et al., 2021), and T2M (Guo et al., 2022a)) with the metrics suggested by Guo et al. (2022a).
As can be seen, MDM achieves state-of-the-art results in
FID , Diversity , and MultiModality , indicating high diversity per input text prompt, and high-quality samples, as can also be seen qualitatively in Figure 1.
User study.
We asked 31 users to choose between MDM and state-of-the-art works in a side-by-side view, with both samples generated from the same text prompt randomly sampled from the KIT test set.
We repeated this process with 10 samples per model and 10 repetitions per sample.
This user study enabled a comparison with the recent TEMOS model (Petrovich et al., 2022), which was not [Figure] included in
Fig.
4 shows that most of the time, MDM was preferred over the compared models, and even preferred over ground samples in of the Method [Table] MultimodalDist
[Table] [Table]
4. A CTION - TO -M OTION
Action-to-motion is the task of generating motion given an input action class, represented by a scalar.
The output motion should faithfully animate the input action, and at the same time be natural and reflect the distribution of the dataset on which the model is trained.
Two dataset are commonly used to evaluate action-to-motion models: HumanAct12 (Guo et al., 2020) and UESTC
(Ji et al., 2018).
We evaluate our model using the set of metrics suggested by Guo et al.
(2020), namely Fr´echet Inception Distance (FID), action recognition accuracy, diversity and multimodality.
The combination of these metrics makes a good measure of the realism and diversity of generated motions. [Table] [Table]
Data.
HumanAct
(Guo et al., 2020) offers approximately 1200 motion clips, organized intoaction categories, with 47 to 218 samples per label.
UESTC
(Ji et al., 2018) consists of 40 action classes, 40 subjects and 25K samples, and is split to train and test.
We adhere to the cross-subject testing protocol used by current works, with 225-345 samples per action class.
For both datasets we use the sequences provided by Petrovich et al.
Implementation.
The implementation presented in Figure 2 holds for all the variations of our work.
In the case of action-to-motion, the only change would be the substitution of the text embedding by an action embedding.
Since action is represented by a scalar, its embedding is fairly simple; each input action class scalar is converted into a learned embedding of the transformer dimension.
The experiments have been run with batch size 64 , a latent dimension of 512 , and an encodertransformer architecture.
Training on HumanAct12 and UESTC has been carried out for 750 K and 2 M steps respectively.
In our tables we display the evaluation of the checkpoint that minimizes the
FID metric.
Quantitative evaluation.
Tables 3 and 4 reflect MDM’s performance on the HumanAct12 and
UESTC datasets respectively.
We conduct 20 evaluations, with 1000 samples in each, and report their average and a 95% confidence interval.
We test two variations, with and without foot contact loss.
Full ablation study for geometric losses is presented in Appendix A.2.
Our model leads the board for both datasets.
The variation with no foot contact loss attains slightly better results; nevertheless, as shown in our supplementary video, the contribution of foot contact loss to the quality of results is important, and without it we witness artifacts such as shakiness and unnatural gestures.
A DDITIONAL A PPLICATIONS 5. M OTION E DITING
In this section we implement two motion editing applications - in-betweening and body part editing , both using the same approach in the temporal and spatial domains correspondingly.
For inbetweening , we fix the first and last 25% of the motion, leaving the model to generate the remaining 50% in the middle.
For body part editing , we fix the joints we don’t want to edit and leave the model to generate the rest.
In particular, we experiment with editing the upper body joints only.
In figure 3 we show that in both cases, using the method described in Section 3 generates smooth motions that adhere both to the fixed part of the motion and the condition (if one was given)
[Table]
5. U NCONSTRAINED S YNTHESIS
The challenging task of unconstrained synthesis has been studied by only a few (Holden et al., 2016;
Raab et al., 2022).
In the presence of data labeling, e.g., action classes or text description, the labels work as a supervising factor, and facilitate a structured latent space for the training network.
The lack of labeling make training more difficult.
The human motion field possesses rich unlabeled datasets (Adobe Systems Inc., 2021), and the ability to train on top of them is an advantage.
Daring to test MDM in the challenging unconstrained setting, we follow MoDi(Raab et al., 2022) for evaluation.
We use the metrics they suggest (FID, KID, precision/recall and multimodality), and run on an unconstrained version of the HumanAct12 (Guo et al., 2020) dataset.
Data.
Although annotated, we use HumanAct12 (see Section 4.2) in an unconstrained fashion, ignoring its labels.
The choice of HumanAct12 rather than a dataset with no labels (e.g., Mixamo (Adobe Systems Inc., 2021)), is for compatibility with previous publications.
Implementation.
Our model uses the same architecture for all forms of conditioning, as well as for the unconstrained setting.
The only change to the structure shown in Figure 2, is the removal of the conditional input, such that z tk is composed of the projection of t only.
To simulate an unconstrained behavior, ACTOR Petrovich et al.
(2021) has been trained by (Raab et al., 2022) with a labeling of one class to all motions.
Quantitative evaluation.
The results of our evaluation are shown in table 5.
We demonstrate superiority over works that were not designed for an unconstrained setting, and get closer to MoDi (Raab et al., 2022).
MoDi is carefully molded for unconstrained settings, while our work can be applied to any (or no) constrain, and also provides editing capabilities.
D ISCUSSION
We have presented MDM, a method that lends itself to various human motion generation tasks.
MDM is an untypical classifier-free diffusion model, featuring a transformer-encoder backbone, and predicting the signal, rather than the noise.
This yields both a lightweight model, that is unburdening to train, and an accurate one, gaining much from the applicable geometric losses.
Our experiments show superiority in conditioned generation, but also that this approach is not very sensitive to the choice of architecture.
A A DDITIONAL E XPERIMENTS A. D IFFUSION P ARAMETERS
First, predicting ϵ instead of X 0 unfortunately proved to not converge throughout our experiments, hence we have no quantifiable outcome to report on this ablation study.
On the other hand [Table] learning MDM with different number of diffusion steps significantly affects performance and holds potential to accelerate inference time.
Table 6 shows optimal performance for T = 100 , in addition, it enables accelerating inference by a factor of 10 compared to T = 1000 , which is widely used for images [Table] A. G EOMETRIC L OSSES We conduct a thorough experiment to evaluate the contribution of geometric losses with the HumanAct12 dataset.
The results are presented in Table 7.
For alignment with prior work, all metrics are calculated using the deep features of the action recognition network suggested by by what?.
In general, MDM scores are too close to the real test distribution ( i.e. the evaluation network fails to discriminate between the two).
This means that quantitative results comparing the different variants
MDM are too similar to evaluate.
As a result, we are not able to decide what combination of geometric losses is preferred.
We leave for future work experimenting with a different, more expressive, evaluation network.
B E VALUATION M ETRICS .
For the completeness of our work, we describe here the quantitative metrics used throughout the paper, as they originally described and implemented by Guo et al.
(2020) for action-to-motion and by Guo et al.
(2022a) for text-to-motion.
[Table]
B. A CTION - TO -M OTION
The following metrics are based on an RNN action recognition network as it was originally trained by Guo et al.
We refer to it as the evaluator network.
Frechet Inception Distance (FID).
A widely used metric to evaluate the overall quality for generation tasks.
FID is calculated upon features extracted from 1,000 generated motion vs ground truth (real) taken from the test set.
To adjust this metric to the motion domain, we extract a deep representation of the motion with the evaluator network instead of the inception neural network
TODO:
CITE, originally used for images.
A lower value implies better FID results.
Accuracy.
We classify 1,000 generated motions using the evaluator network, than we calculate the overall recognition accuracy that indicates the correlation of the motion and its action type.
A higher value implies better accuracy.
Diversity.
Diversity is the metric that measures the variance of the generated motions across all action categories.
For measuring diversity, we first randomly sample two subsets of the same size S d out of a set of all generated motions across all action categories.
Note { v 1 , ..., v S d } and { v ′ 1 , ..., v S ′ d } as the respective sets of motion extracted feature vectors.
The diversity value is considered good if it is close to the diversity of the ground truth.
The diversity of this set of motions is defied as
[Equation]
We define S d = 200 for this experiments.
Multimodality. Unlike diversity, multimodality is the metric to measure the generated motions diversify within each action type. in other words it refers to the interplay between different representational modes. For measuring multimodality, we randomly sample two subsets of the same motion type c of size S l out of a set of motions with C actions types, we extract two subset of veature vectors { v c, 1 , ... , v c,S l } and { v ′ c, 1 , ..., v ′ c,S l } .
The multimodality of this motion set is defined as
Multimodality
[Equation]
We define S l = 20 for this experiments.
B. T EXT - TO -M OTION
Originally suggested by Guo et al.
(2022a), the following metrics are based on a text feature extractor and motion feature extractor jointly trained under contrastive loss to produce geometrically close feature vectors for matched text-motion pairs, and vise versa.
R Precision.
(top-3) For each generated motion, its ground-truth text and a randomly selected missmatched descriptions from the test set.
We calculate the euclidean distance between the motion feature and text feature of each description in the pool.
We count the average accuracy at top3 places.
If the ground truth entry falling into the top-3 candidates, we treat it as True Positive retrieval.
We use a batch size 32 (i.e. 31 negative examples).
FID.
Same as for action-to-motion, using the motion extractor as the evaluator network.
Multimodal Distance.
We calculate the multimodal distance as the average Euclidean distance between the motion feature of each generated motion and the text feature of its corresponding description in test set.
A lower value implies better multimodal distance.
Diversity.
Same as for action-to-motion but with S d = 300 .
Multimodality.
Same as for action-to-motion but with S m = 10 .
C I MPLEMENTATION D ETAILS
The full implementation of MDM could be found in the code to be published.
In addition, the following are the hyperparameters and model details for all of our experiments.
Diffusion framework.
In all of our experiments, we used an implementation of DDPM Ho et al.
(2020) by Dhariwal & Nichol (2021) 4 .
We use T = 1 , 000 diffusion steps, cosine noise scheduling (predefined sigmas).
All other hyperparameters are according to the implementation defaults.
Transformer architecture.
For our transformer architectures, we used the PyTorch implementation 5 .
We used 8 transformer layers, 4 attention heads, latent dimension d = 512 , dropout 0 .
1 , feed-forward size 1024 and gelu activations.
The number of learned parameters for each model is stated in Table 8.
GRU architecture.
We use the PyTorch implementation of GRU (Cho et al., 2014) 6 with two layers and latent dimension 512 .
The number of learned parameters for each model is stated in Table 8.
Learning hyperparameters.
For all of our experiments, we use batch size 64 , learning rate 10 − 4 .
Architecture # Parameters ( · 10 6 )
Transformer Encoder 17 .
Transformer Decoder 26 .+ input token 26 .
U-net
[Table]
D U SER S TUDY
In Section 4.1 we conduct a user study for the text-to-motion task.
We asked 31 users to choose between MDM and state-of-the-art works in a side-by-side view, with both samples generated from the same text prompt randomly sampled from the KIT test set.
We repeated this process withsamples per model and 10 repetitions per sample.
This user study enabled a comparison with the recent TEMOS model (Petrovich et al., 2022), which was not included in the HumanML3D benchmark.
Fig.
4 shows that most of the time, MDM was preferred over the compared models, and even preferred over ground truth samples in 42 .
3% of the cases.
This user study was designed to measure the precision of the models, i.e. which one better fits the input text.
The exact phrasing of the question was “Which animation better fits the following description?”.
A sample question from this study is presented in Fig.