A UTO -C ONDITIONED
LSTM
N ETWORK
FOR
E XTENDED C OMPLEX H UMAN M OTION S YNTHESIS
Anonymous authors
Paper under double-blind review
A BSTRACT
We present a real-time method for synthesizing highly complex human motions using a novel LSTM network training regime we call the auto-conditioned LSTM (acLSTM).
Recently, researchers have attempted to synthesize new motion by using autoregressive techniques, but existing methods tend to freeze or diverge after a couple of seconds due to an accumulation of errors that are fed back into the network.
Furthermore, such methods have only been shown to be reliable for relatively simple human motions, such as walking or running.
In contrast, our approach can synthesize arbitrary motions with highly complex styles, including dances or martial arts in addition to locomotion.
The acLSTM is able to accomplish this by explicitly accommodating for autoregressive noise accumulation during training.
Furthermore, the structure of the acLSTM is modular and compatible with any other recurrent network architecture, and is usable for tasks other than motion.
Our work is the ﬁrst to our knowledge that demonstrates the ability to generate over 18,000 continuous frames (300 seconds) of new complex human motion w.r.t.
different styles.
I NTRODUCTION
The synthesis of realistic human motion has recently seen increased interest (Holden et al., 2016; 2017; Fragkiadaki et al., 2015; Jain et al., 2016;
Bütepage et al., 2017; Martinez et al., 2017) with applications beyond animation and video games.
The simulation of human looking virtual agents is likely to become mainstream with the dramatic advancement of Artiﬁcial Intelligence and the democratization of Virtual Reality.
A challenge for human motion synthesis is to automatically generate new variations of motions while preserving a certain style, e.g., generating large numbers of different Bollywood dances for hundreds of characters in an animated scene of an Indian party.
Aided by the availability of large human-motion capture databases, many database-driven frameworks have been employed to this end, including motion graphs (Kovar et al., 2002; Safonova & Hodgins, 2007;
Min & Chai, 2012), as well as linear (Safonova et al., 2004; Chai & Hodgins, 2005; Tautges et al., 2011) and kernel methods (Mukai, 2011; Park et al., 2002; Levine et al., 2012; Grochow et al., 2004;
Moeslund et al., 2006; Wang et al., 2008), which blend key-frame motions from a database.
It is hard for these methods, however, to add new variations to existing motions in the database while keeping the style consistent.
This is especially true for motions with a complex style such as dancing and martial arts.
More recently, with the rapid development in deep learning, people have started to use neural networks to accomplish this task (Holden et al., 2017; 2016; 2015).
These works have shown promising results, demonstrating the ability of using high-level parameters (such as a walking-path) to synthesize locomotion tasks such as jumping, running, walking, balancing, etc.
These networks do not generate new variations of complex motion, however, being instead limited to speciﬁc use cases.
In contrast, our paper provides a robust framework that can synthesize highly complex human motion variations of arbitrary styles, such as dancing and martial arts, without querying a database.
We achieve this by using a novel deep auto-conditioned LSTM (acLSTM) network architecture.
LSTM networks are autoregressive deep learning frameworks which seek to predict sequences of data similar to a training distribution.
Such a framework is intuitive to apply to human motion, which can be naturally modeled as a time series of skeletal joint positions.
We are not the ﬁrst to leverage
LSTM or RNN networks for this task (Fragkiadaki et al., 2015; Jain et al., 2016; Bütepage et al., 2017;
Martinez et al., 2017), and these works produce reasonably realistic output at a number of tasks such as sitting, talking, smoking, etc.
However, these existing methods also have a critical drawback: the motion becomes unrealistic within a couple of seconds and is unable to recover.
This issue is commonly attributed to error accumulation due to feeding network output back into itself (Holden et al., 2017).
This is reasonable, as the network during training is given ground-truth input sequences to condition its subsequent guess, but at run time, must condition this guess on its own output.
As the output distribution of the LSTM will not be identical to that of the ground-truth, it is in effect encountering a new situation at test-time.
The acLSTM structure compensates for this by linking the network’s own predicted output into its future input streams during training, similar to the technique proposed in (Bengio et al., 2015).
Our method is light-weight and can be used in conjunction with any other LSTM or RNN based learning scheme.
Though straightforward, this technique ﬁxes the issue of error accumulation, and allows the network to output incredibly long sequences without failure, on the order of hundreds of seconds (see Figure 5).
Though we are yet as unable to prove the permanent stability of this structure, it seems empirically that motion can be generated without end.
In summary, we present a new LSTM architecture capable for the ﬁrst time of synthesizing potentially indeﬁnitely long sequences of realistic and complex human motions with respect to different styles.
R ELATED W ORK
Many approaches have developed over the years in order to generate realistic human motion.
In this section, we ﬁrst review the literature that has dealt with motion synthesis using simulation methods and database-driven methods, then review the more recent deep learning approaches.
Simulation-based Methods.
Simulation-based techniques are able to produce physically plausible animations (Levine & Popovi´c, 2012; Clegg et al., 2015; Hämäläinen et al., 2015; Ha et al., 2012), including realistic balancing, motion on terrain of various heights, and recovery from falling.
These techniques tend to consider physical constraints on the human skeleton while optimizing a motion objective.
For example, in the work of Levine et al.
(Levine & Popovi´c, 2012), one task they employ is moving the skeleton in a certain direction without falling over.
Similarly, in Ha et al.
(Ha et al., 2012), given initial fall conditions, they seek to minimize joint stress due to landing impact while ensuring a desired landing pose.
Though the output is plausible and realistic, they require speciﬁc objectives and constraints for each individual task.
It is not feasible to use such explicit objectives or constraints for stylized motion.
The correct arch of the back and ﬂourishes of the limbs for a dance cannot be described a priori and must instead be inferred from the data.
Database-driven Methods.
Motion graphs (Kovar et al., 2002; Safonova & Hodgins, 2007; Min &
Chai, 2012), which stitch transitions into segments queried from a database, can generate locomotion along arbitrary paths, but are in essence limited to producing memorized sequences.
Frameworks based on linear (Safonova et al., 2004; Chai & Hodgins, 2005; Tautges et al., 2011) and kernal methods (Mukai, 2011; Park et al., 2002; Levine et al., 2012; Grochow et al., 2004; Moeslund et al., 2006; Wang et al., 2008) have also shown reasonable success.
Taylor et al.
(Taylor & Hinton, 2009) use conditional restricted boltzmann machines to model motion styles.
Grochow et al.
(Grochow et al., 2004) and Wang et al.
(Wang et al., 2005) both use Gaussian Process Latent Variable Models (GPLVM) to control motion.
Levine et al.
(Levine et al., 2012) apply reinforcement learning in a reduced space to compute optimal motions for a variety of motion tasks, including movement, punching, and kicking.
Kee et al.
(Lee et al., 2010) use a representation of motion data called motion-ﬁelds which allows users to interactively control movement of a character.
The work of Xia et al.
(Xia et al., 2015) is able to achieve real-time style transfers of unlabeled motion.
Taylor et al.
(Taylor et al., 2007) use binary latent variables which are connected between different time steps while Liu et al.
(Liu et al., 2005) estimate physical parameters from motion capture data.
Database methods have limitations in synthesizing new variations of motion, in addition to the high memory costs for storing complex motions.
Deep Learning Approaches.
The use of recurrent networks is a natural approach to dealing with the problem of human motion.
LSTM and RNN networks are trained to generate output sequentially, each output conditioned on the previous elements in the sequence.
These networks have shown much success in Natural Language Processing for generating text (Sutskever et al., 2011), hand written characters (Graves, 2013; Gregor et al., 2015), and even captioning images (Vinyals et al., 2014).
For the purpose of motion prediction and generation, Fragkiadaki et al.
(Fragkiadaki et al., 2015) propose to use encoder-recurrent-decoder (ERD), jointly learning a skeleton embedding along with sequential information, in order to make body positions more realistic.
In Jain et al.
(Jain et al., 2016), the authors employ RNNs to learn spatio-temporal graphs of interaction, a structure which naturally applies to the human skeleton over time–joint positions are related over consecutive frames, and joints also interact with each other spatially (arms and legs interact with each other, the spine interacts with all other joints).
Bütepage et al.
(Bütepage et al., 2017) attempt to learn dance sequentially, but they are unable to produce varied and realistic output.
More recently, Martinez et al.
(Martinez et al., 2017) propose using a sequence-to-sequence architecture along with sampling-based loss.
A main problem with these approaches, even in the case of the ERD, is that motion generation tends to converge to a mean position over time.
Using our proposed training method, motion neither halts nor becomes unrecognizable for any period of time.
Holden et al.
(Holden et al., 2015) demonstrate that a manifold of human motion can be learned using an autoencoder trained on the CMU dataset.
They (Holden et al., 2016) extend this work by using this deep convolutional auto-encoder, in addition to a disambiguation network, in order to produce realistic walking motion along a user-deﬁned path.
They are also able to use the embedding space of the encoder to perform style transfer across different walks.
Using the same autoencoder and a different disambiguation network, they can also perform other user deﬁned tasks such as punching and kicking.
Their method does not use an RNN structure and so it can only generate ﬁxed length sequences based on the architecture of the disambiguation network.
More recently, Holden et al.
(Holden et al., 2017) use a Phase-Functioned Neural Network which takes the geometry of the scene into account to produce motion along a user-deﬁned path.
This method makes use of humans’ periodic change of gait to synthesize realistic walking motion but has not demonstrated the ability to generate motions that have more complex step movements.
M ETHODOLOGY
𝐼 "#$ 𝑆 "#$ 𝑂 " 𝐼 " 𝑆 " 𝑂 "’$ 𝐼 "’$ 𝑆 "’$ 𝑂 "’( 𝐼 "’( 𝑆 "’( 𝑂 "’) 𝑆 "’) 𝑂 "’* 𝑆 "’* 𝑂 "’+ 𝑆 "’+ 𝑂 "’, 𝑆 "’, 𝑂 "’- 𝐼 "’- 𝑆 "’- 𝑂 "’.
𝐼 "’.
𝑆 "’.
𝑂 "’/ 𝐼 "’/ 𝑆 "’/ 𝑂 "’$ 𝐼 "’$ 𝑆 "’$ 𝑂 "’$$ 𝑆 "’$$ 𝑂 "’$(
[Figure]
Auto-Conditioned LSTM Network.
LSTM networks are well documented in the literature.
A good survey and introduction can be found at (Karpathy, 2015; Olah, 2015).
In short, a recurrent network can be represented as a function with a hidden memory unit, x t +1 = f
( x t , m t ) , where m t is the "memory" of the network, and is updated during every forward pass, and initialized at 0.
The motivation is that the memory stores important information about a sequence up until that point, which can help with the prediction of the next element.
An LSTM is a special case of an RNN that has longer-term persistent memory.
As mentioned in the introduction, the major drawback of using LSTM/RNN deep learning methods for motion prediction is the problem of error accumulation.
Following the conventional way to train an RNN, the network is recursively given a sequence of ground truth motion data, G 1 ,k =
[ g 1 , ..., g k ] , and asked to produce the output G 2 ,k +
[ g 2 , ..., g k +1 ] .
Speciﬁcally, at training time, the recursive module at time step t , M t , is conditioned on the input [ g 1 , ..., g t − 1 ] , and the error is measured between its output and g t .
Because of this, the backpropogation algorithm (Williams & Zipser, 1995) used for updating the parameters will always optimize w.r.t.
the input ground-truth sequences [ g 1 , ..., g t − 1 ] .
The parameters are accustomed to ground truth input — something it does not have access to at test-time.
It is easy to see why problems will emerge: even if initial input is similar to ground truth, those slight differences will accumulate as the output is fed back in, producing output that become progressively worse until the sequence diverges or freezes.
Effectively, the network is encountering a completely novel situation during test time as compared to training time, and so cannot perform well.
The issue is so prevalent, in fact, that previous methods fail to produce realistic output after several seconds (Fragkiadaki et al., 2015; Jain et al., 2016;
Martinez et al., 2017).
Interestingly enough, this problem does not emerge in the more popular usages of LSTM networks for NLP problems such as sequential text generation (Sutskever et al., 2011).
Network output for these models is interpreted as probability vectors over characters, the output character being the entry with the highest probability.
It is easy to round these to resemble ground-truth instances by setting the output character value to 1 and the other to 0, so there is no major difference between training and test time input.
One might imagine a similar approach to deal with motion.
A naive solution could be to ﬁx the length of the bones before throwing it back in, for example, but it does not guarantee against other failure cases, such as impossible angles of joints.
Holden et al.
(Holden et al., 2015) show that an autoencoder framework can to some degree "ﬁx" broken input, and some researchers have tried jointly learning such an encoder-decoder network alongside RNNs to better condition subsequent input (Gregor et al., 2015;
Fragkiadaki et al., 2015).
However, in the case of this framework being applied to motion as in ERD (Fragkiadaki et al., 2015), it does not generalize to indeﬁnitely long sequences, as shown in (Jain et al., 2016).
It seems as though the autoencoder might mitigate error accumulation, but does not eliminate it.
The acLSTM, on the other hand, deals with poor network output explicitly by using it during training.
Instead of only feeding in ground-truth instances, we use subsequences of the network’s own outputs at periodic intervals.
For instance, sticking with the example above, instead of conditioning the network on G 1 ,k
= [ g 1 , ..., g k ] , we use ˆ G 1 ,k
= [ g 1 , ..., g u , p u +1 , ..., p u + v , g u + v +1 .., g k ] to predict
G 2 ,k +1 =
[ g 2 , ..., g k +1 ] .
The variable p u +1 is the network output conditioned on [ g 1 , ..., g u ] , and p u +2 is conditioned on [ g 1 , ..., g u , p u +1 ] .
In this example, we refer to v as the "condition length" and u as the "ground-truth length".
As the network is conditioned on its own output during training, it is able to deal with such input during synthesis.
Figure 1 details an unrolled Auto-Conditioned RNN with condition length u
= v = 1 , and Figure 10 shows a more detailed view or our network.
The method of (Bengio et al., 2015) also proposes using network output during training, but does so stochastically, without ﬁxing condition lengths.
However, we found that changing the condition/ground-truth length while keeping the proportion of ground-truth input ﬁxed affects both the accuracy and variation of the output.
See Figure 9 in the appendix.
Auto-conditioning also has the interpretation of training the network to produce longer sequences without further input.
Whereas with standard training the network error is measured only against p u +1 when conditioned with [ g 1 , ..., g u ] , under auto-conditioning the error is computed on the entire sequence p u +1 , ..., p u + v w.r.t.
the same input.
This effectively forces the network to produce v frames of output simultaneously as opposed to only one.
Martinez et al.
(Martinez et al., 2017) also use contiguous sequences of network output during training, but unlike us they do not alternate these with ground-truth input at regular intervals.
Data Representation.
We use the publicly available CMU motion-capture dataset for our experiments.
The dataset is given as sequences of 57 skeleton joint positions in 3D-space.
First, we deﬁne a root joint, whose position at time t is given by r t = ( r 1 ,t , r 2 ,t , r 3 ,t ) .
In order to better capture relative motion, we instead use the displacement of the root from the previous frame for input — ˜ r t = ( r 1 ,t − r 1 ,t − 1 , r 2 ,t − r 2 ,t − 1 , r 3 ,t − r 3 ,t − 1 ) .
For every other joint at time t , with position j t = ( j 1 ,t , j 2 ,t , j 3 ,t ) , we represent it as as the relative distance in the world-coordinate system to the root joint, ¯ j t =
( j 1 ,t − r 1 ,t , j 2 ,t − r 2 ,t , j 3 ,t − r 3 ,t ) .
All distances are stored in meters.
We use a skeleton with height 1.54 meters in neutral pose for all experiments.
We found this representation to be desirable for several reasons.
Primarily, if there is periodic motion in the dataset, we would like frames at the same point in the repeated activity to have small Euclidean distance.
If we instead used absolute positions, even if it were only for the hip, this would certainly not be the case.
We note that there are alternative representations which achieve the same property.
(Fragkiadaki et al., 2015) express joint positions as rotations relative to a parent joint, and (Holden et al., 2016) deﬁne them in the body’s relative coordinate system along with a relative rotation of the body w.r.t.
the previous frame.
Training.
We train the acLSTM with three fully connected layers with a memory size of 1024, similar to (Fragkiadaki et al., 2015; Jain et al., 2016; Bütepage et al., 2017).
The main difference is that for every u ground-truth inputs of the time series, we connect v instances of the network’s own output into its subsequent input streams (see section 2.1).
In the main body of the paper, we set u = v = 5 .
We carry out further experiments with varying u and v in the appendix.
We train with a sequence length of 100 for 500000 iterations using the ADAM backpropogation algorithm (Kingma & Ba, 2014) on an NVIDIA 1080 GPU for each dataset we experiment on.
We use Euclidean loss for our objective function.
The initial learning rate is is set to 0.0001.
We implement the training using the python caffe framework (Jia et al., 2014).
We sample sequences at multiple frame-rates as well as rotate and translate the sequence randomly in order to increase the training size.
In detail, if at time t we input the network H with ground truth, then the loss is given by:
[Equation]
where x t and x t +1 are the ground truth motions for time steps t and t + 1 .
If at time t the network is input with its own previous output, the loss is given by:
[Equation]
where ˆ x tk = H(H
( ... H( x t − k , m t − k ) , m t − 1 ) , m t ) .
k indicates how many times the network has fed itself is its own input since the last injection of ground-truth.
It is bounded by the condition length (see previous section) [Table]
E XPERIMENTAL R ESULTS
We evaluate our synthesized motion results on different networks trained on each of four distinct subsets from the CMU motion capture database: martial arts, Indian dance, Indian/salsa hybrid
[Table]
and walking.
An anonymous video of the results can be found here: https://youtu.be/FunMxjmDIQM.
[Table]
Quantitative Results.
Table 1 shows the prediction error as Euclidean distance from the ground truth for different motion styles at various time frames.
 
        : D O N L QJ /670/ DF/670/5ZFOHQJWK 


        , QG L DQ  ' DQ F H
 
     )UDPH1XPEHU     0 D U W L D O  $ U W V
[Figure]
We compare with a 3-layer LSTM (LSTM-3LR), as well as ERD (Fragkiadaki et al., 2015) and the seq2seq framework of (Martinez et al., 2017).
Though our goal is long-term stable synthesis and not prediction, it can be seen in Table 1 that acLSTM performs reasonably well in this regard, achieving the best performance at 640 ms in both the Indian Dance and Martial Arts categories.
The method of (Martinez et al., 2017) performs the best in the short term, but has the worst error by far past the half-second mark.
As noted in (Fragkiadaki et al., 2015), the stochasticity of human motion makes longer-term prediction infeasible.
Table 2 in the appendix shows this error difference between versions of the Indian dance network trained with different condition lengths.
Figureshows the average change between subsequent frames for different frame times for the acLSTM and the basic scheme.
For the more complex motions of Martial Arts and Indian Dance, it is clear from the graph that acLSTM continues producing motion in the long-term while the basic training scheme (LSTM-3LR) without auto-conditioning results in stagnated motion, as the network freezes into a converged mean position.
Likewise, Figure 9 in the appendix shows this average change for the Indian dance network trained with different condition lengths.
We note that while the methods of (Fragkiadaki et al., 2015; Martinez et al., 2017) do not simply freeze completely, as with the basic scheme, their motion becomes unrealistic at around the same time (Figure 7).
This is consistent with the pbservations of the original authors.
[Figure]
Qualitative Results.
Figure 4 shows several example frames taken from 50 second synthesized outputs, representing both the extended long term complexity and plausibility of the output.
In comparison, our implementations of ERD and seq2seq are only able to generate motion for a couple of seconds before they become unrealistic (Figure 7).
We also demonstrate the possibility of creating hybrid motions by mixing training sets in third row of Figure 4.
It should be noted that the motion in our framework, while never permanently failing, also does not remain perfectly realistic, and short-term freezing does sometimes occur.
This, however, does not occur for the network trained just on walking.
It is perhaps that the movement of the feet in conjunction with the absolute movement of the root joint is not as easy to discern when the feet leave the ground aperiodically, or there are jumping motions.
When motion does stagnate, it recovers relatively quickly, and the motion never diverges or freezes completely (see Figure 6).
The short-term freezing could possibly be explained by "dead-times" in the training sequence, where the actor is beginning or ending a sequence which involves a rest position.
Note that when training with two different datasets, as in the case of the Indian/Salsa combined [Figure] [Figure] network, motion borrows from both dance styles.
We also demonstrate in Figure 5 that our method does not freeze even after 20,000 frames of synthesis, which is approximately 333 seconds of output.
One can see a qualitative comparison of acLSTM with a basic LSTM-3R in Figure 3, both trained on the Indian dance dataset.
We ﬁnd the performance of the vanilla network to be consistent with the results reported in (Fragkiadaki et al., 2015; Jain et al., 2016; Bütepage et al., 2017; Martinez et al., 2017), freezing at around 1000 ms.
It never recovers from the motion.
Our network, on the other hand, continues producing varied motion for the same time frame.
D ISCUSSION AND F UTURE W ORK
We have shown the effectiveness of the novel acLSTM architecture to produce extended sequences of complex human motion.
We believe our work demonstrates qualitative state-of-the-art results in motion generation, as all previous work has focused on synthesizing relatively simple human motion for extremely short time periods.
These works demonstrate motion generation up to a couple of seconds at most while acLSTM does not fail even after over 300 seconds.
Though we are as of yet unable to prove indeﬁnite stability, it seems empirically that acLSTM can generate arbitrarily long sequences.
It should be noted that our method is compatible with any other autoregressive training procedure, including (Fragkiadaki et al., 2015; Jain et al., 2016; Martinez et al., 2017), for further improvement.
Current problems that exist include choppy motion at times, self-collision of the skeleton, and unrealistic sliding of the feet.
This latter issue seems to effect only one foot at a time.
In this vein, we believe combining our technique with physically based simulation to ensure realism after synthesis is a natural next step.
Finally, it is important to study the effects of using various condition lengths during training.
We begin the exploration of this topic in the appendix, but further analysis is needed.
A T HE E FFECT OF C ONDITION L ENGTH


     )UDPH1XPEHU          FOHQJWK 
[Figure]
[Table]
B P REDICTION E RROR
It seems that Table 2 and Figure 9 might imply some sort of trade off between motion change over time and short-term motion prediction error when training with different condition lengths.
However, it is also possible that limiting motion magnitude on this particular dataset might correspond to lower error.
Further experiments of various condition lengths on several motion styles need to be conducted to say anything meaningful about the effect
C V ISUAL D IAGRAM OF A UTO -C ONDITIONED LSTM
𝝈 𝝈 𝒕𝒂𝒏𝒉 𝝈
X
X
X + 𝑰 𝒕− tanh 𝑶 𝒕− 𝝈 𝝈 𝒕𝒂𝒏𝒉 𝝈
X
X
X + tanh 𝑶 𝒕 𝝈 𝝈 𝒕𝒂𝒏𝒉 𝝈
X
X
X + 𝑰 𝒕 tanh 𝑶 𝒕+