D ANCE R EVOLUTION : L ONG -T ERM D ANCE G ENERATION WITH M USIC VIA C URRICULUM L EARNING
A BSTRACT
Dancing to music is one of human’s innate abilities since ancient times.
In machine learning research, however, synthesizing dance movements from music is a challenging problem.
Recently, researchers synthesize human motion sequences through autoregressive models like recurrent neural network (RNN).
Such an approach often generates short sequences due to an accumulation of prediction errors that are fed back into the neural network.
This problem becomes even more severe in the long motion sequence generation.
Besides, the consistency between dance and music in terms of style, rhythm and beat is yet to be taken into account during modeling.
In this paper, we formalize the music-conditioned dance generation as a sequence-to-sequence learning problem and devise a novel seq2seq architecture to efﬁciently process long sequences of music features and capture the ﬁne-grained correspondence between music and dance.
Furthermore, we propose a novel curriculum learning strategy to alleviate error accumulation of autoregressive models in long motion sequence generation, which gently changes the training process from a fully guided teacher-forcing scheme using the previous ground-truth movements, towards a less guided autoregressive scheme mostly using the generated movements instead.
Extensive experiments show that our approach signiﬁcantly outperforms the existing state-of-the-arts on automatic metrics and human evaluation.
We also make a demo video to demonstrate the superior performance of our proposed approach at https://www.youtube.com/watch?v=lmE20MEheZ8 .
Arguably, dancing to music is one of human’s innate abilities, as we can spontaneously sway along with the tempo of music we hear.
The research in neuropsychology indicates that our brain is hardwired to make us move and synchronize with music regardless of our intention (Chen et al., 2008).
Another study of archaeology also suggests that dance is a social communication skill among early humans connected to the ability of survival long time ago (Adshead-Lansdale & Layson, 2006).
Nowadays, dance (to music) has become a means to the cultural promotion, a method of emotional expression, a tool for socialization and an art form to bring aesthetic enjoyment.
The neurological mechanism behind dancing behavior and the unique value of dance to society motivate us to explore a computational approach to dance creation from a piece of music in artiﬁcial intelligence research.
Such work is potentially beneﬁcial to a wide range of applications such as dance creation assistant in art and sports, character motion generation for audio games and research on cross-modal behavior.
In literature, the music-conditioned dance generation is a relatively new task that attracts increasing research interests recently.
Early works (Fan et al., 2011; Lee et al., 2013) synthesize dance sequences from music by retrieval-based methods, which show the limited creativity in practice.
Recently, Lee et al.
(2019) formulate the task from the generative perspective, and further propose a decompositionto-composition framework.
Their model ﬁrst generates basic dance units from the music clips and then composes them by using the last pose of current unit to initialize the ﬁrst pose of the next unit.
Although this approach shows better performance than the retrieval-based methods, several challenges still remain.
First, existing generative methods synthesize new human motion sequences through autoregressive models like RNN, which tend to result in short sequences.
In other words, the generated sequences often quickly freeze within a few seconds due to an accumulation of prediction errors that are fed back into the neural network.
This problem becomes even more severe in long motion sequence generation.
For instance, composing a dance for a 1-minute music clip under 15 Frame Per Second (FPS) means generating 900 poses at one time.
In practice, we need novel methods which can effectively generate long motion sequences.
Besides, how to enhance the harmony between the synthesized dance movements and the given music is a largely unexplored challenge.
Inutitively, the movements need to be consistent with the music in terms of style, rhythm and beat.
However, to achieve this goal is non-trivial, which requires the generation model to have the capability to capture the ﬁne-grained correspondence between music and dance.
In this paper, we formalize music-conditioned dance generation as a sequence-to-sequence learning problem where the ﬁne-grained correspondence between music and dance is represented through sequence modeling and their alignment is established via mapping from a sequence of acoustic features of the music to a sequence of movements of the dance.
The model consists of a music encoder and a dance decoder.
The encoder transforms low-level acoustic features of an input music clip into high-level latent representations via self-attention with a receptive ﬁeld restricted within k -nearest neighbors of an element.
Thus, the encoder can efﬁciently process long sequences of music features, e.g., a sequence with more than 1000 elements, and model local characteristics of the music such as chord and rhythm patterns.
The decoder exploits a recurrent structure to predict the dance movement frame by frame conditioned on the corresponding element in the latent representations of music feature sequence.
Furthermore, we propose a curriculum learning (Bengio et al., 2009) strategy to alleviate error accumulation (Li et al., 2017) of autoregressive models in long motion sequence generation.
Speciﬁcally, it gently changes the training process from a fully guided teacher-forcing scheme using the previous ground-truth movements, towards a less guided autoregressive scheme which mostly utilizes the generated movements instead.
This strategy bridges the gap between training and inference of autoregressive models, and thus alleviates error accumulation at inference.
The length of each video clip in the dataset released by Lee et al.
(2019) is only about 6 seconds, which cannot be used by other methds to generate long-term dance except for their specially designed decomposition-to-composition framework.
To facilitate the task of long-term dance generation with music, we collect a high-quality dataset consisting of 1-minute video clips, totaling about 12 hours.
And there are three representative styles in our dataset: “Ballet”, “Hiphop” and “Japanese Pop”.
Our contributions in this work are four-fold: (1) We formalize music-conditioned dance generation as a sequence-to-sequence learning problem and devise a novel seq2seq architecture for the long-term dance generation with music.
(2) We propose a novel curriculum learning strategy to alleviate error accumulation of autoregressive models in long motion sequence generation.
(3) To facilitate long-term dance generation with music, we collect a high-quality dataset that is available with our code 1 .
The extensive experiments show that our approach signiﬁcantly outperforms the existing state-of-the-arts on both automatic metrics and human judgements.
The demo video in the supplementary material also exhibits our approach can generate diverse minute-length dances that are smooth, natural-looking, style-consistent and beat-matching with the musics from test set.
R ELATED W ORK
Cross-Modal Learning.
Most existing works focus on the modeling between vision and text, such as image captioning (Lu et al., 2017;
Xu et al., 2015) and text-to-image generation (Reed et al., 2016; Zhang et al., 2017).
There are some other works to study on the translation between audio and text like Automatic Speech Recognition (ASR) (Hinton et al., 2012) and Text-To-Speech (TTS) (Oord et al., 2016).
While the modeling between audio and vision is largely unexplored and the music-conditioned dance generation is a typical cross-modal learning problem from audio to vision.
Human Motion Prediction.
Prediction of human motion dynamics has been a challenging problem in computer vision, which suffers from the high spatial-temporal complexity.
Existing works (Chan et al., 2019; Wang et al., 2018) represent the human pose as 2D or 3D body keyjoints (Cao et al., 2017) and address the problem via sequence modeling.
Early methods, such as hidden markov models (Lehrmann et al., 2014), Gaussian processes (Wang et al., 2006) and restricted boltzmann machines (Taylor et al., 2007), have to balance the model capacity and inference complexity due to complicated training procedures.
Recently, neural networks dominate the human motion modeling.
For instance, Fragkiadaki et al.
(2015) present LSTM-3LR and Encoder-Recurrent-Decoder (ERD)
as two recurrent architectures for the task
; Jain et al.
(2016) propose a structural-RNN to model human-object interactions in a spatio-temporal graph; and Ghosh et al.
(2017) equip LSTM-3LR with a dropout autoencoder to enhance the long-term prediction.
Besides, convolutional neural networks (CNNs) have also been utilized to model the human motion prediction (Li et al., 2018).
Audio-Conditioned Dance Generation.
In the research of audio-conditioned dance generation, most existing works study on the 2D dance motion generation with music since the training data for paired 2D pose and music can be extracted from the huge amount of dance videos available online.
Various methods have been proposed to handle this task, such as adversarial learning based models (Lee et al., 2019; Sun et al., 2020; Ferreira et al., 2021), autoencoder models (Tang et al., 2018) and sequence-to-sequence models (Lee et al., 2018;
Ren et al., 2019; Yalta et al., 2019;
Ye et al., 2020).
While these works mainly focus on exploring the different neural architectures and overlook the freezing motion issue in dance motion synthesis.
In this work, we ﬁrst propose a novel seq2seq architecture to model the ﬁne-grained correspondence between music and dance, then introduce a novel curriculum learning strategy to address the freezing motion issue caused by error accumulation (Li et al., 2017) in long-term dance motion generation.
A PPROACH
In this section, we present our approach to music-conditioned dance generation.
After formalization of the problem in question, we elaborate the model architecture and the dynamic auto-condition learning approach that facilitates long-term dance generation according to the given music.
[Figure]
3. P ROBLEM F ORMALIZATION
Suppose that there is a dataset D
= { ( X i , Y i ) } Ni =1 , where X = { x t } nt =1 is a music clip with x t being a vector of acoustic features at time-step t , and Y = { y t } n t
=1 is a sequence of dance movements with y t aligned to x t .
The goal is to estimate a generation model g ( · ) from D , and thus given a new music input X , the model can synthesize dance Y to music X based on g ( X ) .
We ﬁrst present our seq2seq architectures chosen for music-conditioned dance generation in the following section.
Later in the experiments, we empirically justify the choice by comparing the architectures with other alternatives.
M ODEL A RCHITECTURE
In the architecture of g ( · ) , a music encoder ﬁrst transforms X =
( x 1 , ..., x n ) ( x i ∈ R d x ) into a hidden sequence Z = ( z 1 , ..., z n ) ( z i ∈ R d z ) using a local self-attention mechanism to reduce the memory requirement for long sequence modeling, and then a dance decoder exploits a recurrent structure to autoregressively predicts movements Y =
( y 1 , ..., y n ) conditioned on Z .
Music Encoder.
Encouraged by the compelling performance on music generation (Huang et al., 2018), we deﬁne the music encoder with a transformer encoder structure.
While the self-attention mechanism (Vaswani et al., 2017) in transformer can effectively represent the multi-scale structure of music, the quadratic memory complexity O ( n 2 ) about sequence length n impedes its application to long sequence modeling due to huge of GPU memory consumption (Child et al., 2019).
To keep the effectiveness of representation and control the cost, we introduce a local self-attention mechanism that modiﬁes the receptive ﬁeld of self-attention by restricting the element connections within k -nearest neighbors.
Thus, the memory complexity is reduced to O ( nk ) .
k could be small in our scenario since we only pursue an effective representation for a given music clip.
Therefore, the local patterns of music are encoded in z t that is sufﬁcient for the generation of movement y t at time-step t , which is aligned with the common sense that the dance movement at certain time-step is highly inﬂuenced by the nearby clips of music.
Yet we can handle long sequences of acoustic features, e.g., more than 1000 elements, in an efﬁcient and memory-economic way.
Speciﬁcally, we ﬁrst embed X = ( x 1 , ..., x n ) into U =
( u 1 , ..., u n ) with a linear layer parameterized by W E ∈ R d x × d z .
Then ∀ i ∈ { 1 , ..., n } , z i can be formulated as:
[Equation]
where F ( · ) : R d v → R d z is a feed forward neural network.
Each u j is only allowed to attend its k -nearest neighbors including itself, where k is a hyper-parameter referring to the sliding window size of the local self-attention.
Then attention weight α ij is calculated using a softmax function as:
[Equation]
where for the the l -th head, W Ql , W Kl ∈ R d z × d k and W Vl ∈ R d z × d v are parameters that transform
U into a query , a key , and a value respectively.
d z is the dimension of hidden state z
i while d k is the dimension of query , key and d v is the dimension of value .
Dance Decoder.
We choose a recurrent neural network as the dance decoder in consideration of two factors: (1) the chain structure can well capture the spatial-temporal dependency among human movement dynamics, which has proven to be highly effective in the state-of-the-art methods for human motion prediction (Li et al., 2018; Mao et al., 2019); (2) our proposed learning strategy is tailored for the autoregressive models like RNN, as will be described in the next section.
Speciﬁcally, with Z = ( z 1 , ..., z n ) , the dance movements Y =
(ˆ y 1 , ..., ˆ y n ) are synthesized by:
[Equation]
where h i is the i -th hidden state of the decoder and h 0 is initialized by sampling from the standard normal distribution to enhance variation of the generated sequences.
[ · ; · ] denotes the concatenation operation.
W S ∈ R ( d s + d z ) ×
d y and b ∈ R d y are parameters where d s and d y are the dimensions of h i and ˆ y i , respectively.
At the i -th time-step, the decoder predicts the movement
ˆ y i conditioned on h i as well as the latent feature representation
z i , and thus can capture the ﬁne-grained correspondence between music feature sequence and dance movement sequence.
3. D YNAMIC A UTO -C ONDITION L EARNING A PPROACH
Exposure bias (Ranzato et al., 2015; Lamb et al., 2016; Zhang et al., 2019) is a notorious issue in natural language generation (NLG), which refers to the train-test discrepancy of autoregressive generative models.
This kind of models are never exposed to their own prediction errors at the training phase due to the conventional teacher-forcing training scheme.
However, compared to NLG, motion sequence generation suffers from the more severe exposure bias problem.
In NLG, the bias of predicted probability distribution over vocabulary can be corrected by sampling strategy.
For instance, we can still generate the target token with index 2 by sampling on probability distribution [0.3, 0.3, 0.4] whose groundtruth is [0, 0, 1].
While any small biases of the predicted motion at each time-step will be accumulated and propagated to the future, since each generated motion is a real-valued vector (rather than a discrete token) representing 2D or 3D body keyjoints in the continuous space.
As a result, the generated motion sequences tend to quickly freeze within a few seconds.
This problem becomes even worse in the generation of long motion sequences, e.g., more than 1000 movements.
Scheduled sampling (Bengio et al., 2015) is proposed to alleviate exposure bias in NLG, which does not work in long motion sequence generation.
Since its sampling-based strategy would feed long predicted sub-sequences of high bias into the model at early stage, which causes gradient vanishing at training due to error accumulation of these high-biased sub-sequences.
Motivated by this, we propose a dynamic auto-condition training method as a novel curriculum learning strategy to alleviate the error accumulation of autoregressive models in long motion sequence generation.
Speciﬁcally, our learning strategy gently changes the training process from a fully guided teacher-forcing scheme using the previous ground-truth movements, towards a less guided autoregressive scheme mostly using the generated movements instead, which bridges the gap between training and inference.
As shown in Figure 1, The decoder predicts Y tgt
=
{ y 1 , ..., y p , y p +1 , ..., y p + q , y p + q +1 , ... } from
Y in = { y 0 , ˆ y 1 , ..., ˆ y p , y p +1 , ..., y p + q , ˆ y p + q
+1 , ... } which alternates the predicted sub-sequences with length p
(e.g., ˆ y 1 , ..., ˆ y p ) and the ground-truth sub-sequences with length q (e.g., y p
+1 , ..., y p + q ).
yis a predeﬁned begin of pose (BOP).
During the training phase, we ﬁx q and gradually increase p according to a growth function f ( t ) ∈ { const, (cid:98) λt (cid:99) , (cid:98) λt 2 (cid:99) , (cid:98) λe t (cid:99)} where t is the number of training epochs and λ < 1 controls how frequently p is updated.
We choose f ( t ) =
(cid:98) λt (cid:99) in practice through the empirical comparison.
Note that our learning strategy degenerates to the method proposed in Li et al.
(2017) when f ( t ) = const .
While the advantage of a dynamic strategy over a static strategy lies in that the former introduces the curriculum learning spirit to dynamically increase the difﬁculty of curriculum (e.g., larger p ) as the model capacity improves during training, which can further improve the model capacity and alleviate the error accumulation of autoregressive predictions.
Finally, we estimate parameters of g ( · ) by minimizing the (cid:96) 1 loss on D :
[Equation]
E XPERIMENTAL S ETUP 4. D ATASET C OLLECTION AND P REPROCESSING
Although the dataset (Lee et al., 2019) contains about 70-hour video clips, the average length of each clip is only about 6 seconds.
Moreover, their dataset cannot be used by other methods to generate longterm dance except for their specially designed decomposition-to-composition framework.
To facilitate the task of long-term dance generation with music, we collect a high-quality dataset consisting of 790 one-minute clips of pose data with 15 FPS, totaling about 12 hours.
And there are three styles in the dataset including “Ballet”, “Hiphop” and “Japanese Pop”.
Table 1 shows the statistics of our dataset.
In the experiment, we randomly split the dataset into the 90% training set and 10% test set.
Pose Preprocess.
For the human pose estimation, we leverage OpenPose (Cao et al., 2017) to extract 2D body keyjoints from videos.
Each pose consists of 25 keyjoints 2 and is represented by a 50-dimension vector in the continuous space.
In practice, we develop a linear interpolation algorithm to ﬁnd the missing keyjoints from nearby frames to reduce the noise in extracted pose data.
[Table]
Audio Preprocess.
Librosa (McFee et al., 2015) is a well-known audio and music analysis library in the music information retrieval, which provides ﬂexible ways to extract the spectral and rhythm features of audio data.
Speciﬁcally, we extract the following features: mel frequency cepstral coefﬁcients (MFCC) , MFCC delta , constant-Q chromagram , tempogram and onset strength .
To better capture the beat information of music, we convert onset strength into a one-hot vector as an additional feature to explicitly represent the beat sequence of music, i.e., beat one-hot .
Finally, we concatenate these audio features as the representation of a music frame, as shown in Table 4 in Section 6.2.
During extracting audio features, the sampling rate is 15,400Hz and hop size is 1024.
Hence, we haveaudio samples per second that is aligned with the 15 FPS of pose data.
4. I MPLEMENTATION D ETAILS
The music encoder consists of a stack of N = 2 identical layers.
Each layer has two sublayers: a local self-attention sublayer with l = 8 heads and a position-wise fully connected feed-forward sublayer with 1024 hidden units.
Each head contains a scaled dot-product attention layer with d k = d v =and its receptive yield is restricted by setting k = 100 .
Then we set the sequence length n = 900 , dimension of acoustic feature vector d x = 438 , dimension of hidden vector d z = 256 , respectively.
The dance decoder is a 3-layer LSTM with d s = 1024 and the dimension of pose
vector d y = 50 .
We set λ = 0 .
01 and q = 10 for the proposed learning approach and train the model using the Adam optimizer with the learning rate 1e-4 on 2 NVIDIA V100 GPUs.
4. B ASELINES
The music-conditioned generation is an emerging task and there are few methods proposed to solve this problem.
In our experiment, we compare our proposed approach to the following baselines: (1) Dancing2Music.
We use Dancing2Music (Lee et al., 2019) as the primary baseline that is the previous state-of-the-art; (2) Aud-MoCoGAN.
Aud-MoCoGAN is an auxiliary baseline used in
Dancing2Music, which is original from MoCoGAN (Tulyakov et al., 2018); (3) LSTM.
Shlizerman et al.
(2018) propose a LSTM network to predict body dynamics from the audio signal.
We modify their open-source code to take audio features of music as inputs and produce human pose sequences.
E XPERIMENTAL R ESULTS
In this section, we conduct extensive experiments to evaluate our approach and compare it to the aforementioned baselines.
Note that we do not include comparison experiment on the dataset collect by Lee et al.
due to the above issue in Section 4.1.
5. A UTOMATIC M ETRICS AND H UMAN E VALUATION
We evaluate the different methods by automatic metrics and conduct a human evaluation on motion realism and smoothness of generated dance movements, as well as style consistency.
Speciﬁcally, we randomly sample 60 music clips from the test set to generate dances, then divide the dances generated by three methods and the real dances into four pairs: (LSTM, ours), (Dancing2Music, ours), (real, ours) and (real, real).
We invite 10 amateur dancers as the annotators to answer 3 questions for each pair that is blind for them (use (A, B) pair instead): (1) Which dance is more realistic regardless of music?
(2) Which dance is more smooth regardless of music?
(3) Which dance matches the music better with respect to style?
Note that, we do not include Aud-MoCoGAN in the human evaluation, since both it and Dancing2Music are GAN based methods while the latter is the state-of-the-art.
[Table]
We evaluate the visual quality and realism of generated dances by Fréchet Inception Distance (FID) (Heusel et al., 2017), which is used to measure how close the distribution of generated dances is to the real.
Similar to Lee et al.
(2019), we train a style classiﬁer on dance movement sequences of three styles and use it to extract features for the given dances.
Then we calculate FID between the synthesized dances and the real.
As shown in
Table 2, our FID score is signiﬁcantly lower than those of baselines and much closer to that of the real, which means our generated dances are more motion-realistic and more likely to the real dances.
Besides, we use the same classiﬁer to measure the style accuracy of generated dance to the music, and our approach achieves 77.6% accuracy and signiﬁcantly outperforms Dancing2Music by 17.2%.
Human evaluation in Figure 2 consistently shows the superior performance of our approach, compared to baselines on motion realism, style consistency and smoothness.
We observe the dances generated by LSTM have lots of ﬂoating movements and tend to quickly freeze within several seconds due to the error accumulation, which results in lower preferences.
While Dancing2Music can generate the smooth dance units, the synthesized dances still have signiﬁcant jumps where dance units are connected due to the inherent gap between them, which harm its overall performance in practice.
This is also reﬂected in the preference comparisons to our approach, in which only 35% annotators prefer Dancing2Music on motion realism and 21.7% prefer Dancing2Music on smoothness.
Besides, the result on style consistency shows our approach can generate more style-consistent dances with musics.
Since our approach models the ﬁne-grained correspondence between music and dance at the methodological level and also introduces more speciﬁc music features.
However, Dancing2Music only considers a global style feature extracted by a pre-trained classiﬁer in the dance generation.
In the comparison to real dances, 41.2% annotators prefer our method on motion realism while 30.3% prefer on style consistency.
Additionally, we found that 57.9% annotators prefer our approach on smoothness compared to real dances.
Since the imperfect OpenPose (Cao et al., 2017) introduces the minor noise on pose data extracted from dance videos.
On the other hand, in the preprocessing, we develop an interpolation algorithm to ﬁnd missing keyjoints and reduce the noise of training pose data.
Besides, this also indicates the chain structure based decoder can well capture the spatial-temporal dependency among human motion dynamics and produce the smooth movements.
Beat Coverage and Hit Rate.
We also evaluate the beat coverage and hit rate introduced in Lee et al.
(2019), which deﬁnes the beat coverage as B k /B m and beat hit rate as B a /B k , where B k is the number of kinematic beats, B m is the number of musical beats and B a is the number of kinematic beats that are aligned with the musical beats.
An early study (Ho et al., 2013) reveals that kinematic beat frames occur when the direction of the movement changes.
Therefore, similar to Yalta et al.
(2019), we use the standard deviation (SD) of movement to detect the kinematic beats in our experiment.
The onset strength (Ellis, 2007) is a common way to detect musical beats.
Figureshows two short clips of motion SD curves and aligned musical beats.
We observed that the kinematic beats occur where the musical beats occur, which is consistent with the common sense that dancers would step on musical beats during dancing but do not step on every musical beat.
As we can see in
Table 2, LSTM and Aud-MoCoGAN generate dances with few kinematic beats and most of them do not match the musical beats.
Our method outperforms Dancing2Music by 6.1% on beat coverage and 2.7% on hit rate, which indicates that introducing the features about musical beat into model is beneﬁcial to the better beat alignment of generated dance and input music.
Diversity and Multimodality.
Lee et al.
(2019) introduce the diversity metric and the multimodality metric.
The diversity metric evaluates the variations among generated dances corresponding to various music, which reﬂects the generalization ability of the model and its dependency on the input music.
Similarly, we use the average feature distance as the measurement and these features are extracted by the same classiﬁer used in measuring FID.
We randomly sample 60 music clips from test set to generate dances and randomly pick 500 combinations to compute the average feature distance.
Results in Table 2 show the performance of our approach is superior to Dancing2Music on diversity.
The reason is that Dancing2Music only considers a global music style feature in generation while different music of the same style have the almost same style feature.
Our approach models the ﬁne-grained correspondence between music and dance at the methodological level, simultaneously takes more speciﬁc music features into account.
In other words, our approach is more dependent on input music and thus can generate different samples for difference music clips of the same style.
Multimodality metric evaluates the variations of generated dances for the same music clip.
Speciﬁcally, we generate 5 dances for each of randomly sampled 60 music clips and compute the average feature distance.
As we can see, our approach slightly underperforms Dancing2Music and AudMoCoGAN since it has more dependency on music inputs as aforementioned, which make the generated dances have the limited variation patterns given the same music clip.
5. L ONG -T ERM D ANCE G ENERATION
To further investigate the performance of different methods on long-term dance generation, we evaluate FID of dances generated by different methods over time.
Speciﬁcally, we ﬁrst split a generated dance with 1 minute into 15 four-second clips and measure FID of each clip.
Figure 4 shows
FID scores of LSTM and Aud-MoCoGAN grow rapidly in the early stage due to error accumulation and converge to the high FID, since the generated dances quickly become frozen.
Dancing2Music maintains the relatively stable FID scores all the time, which beneﬁts from its decomposition-tocomposition method.
While its curve still has subtle ﬂuctuations since it synthesizes whole dances by
[Figure]
[Table]
composing the generated dance units.
Compared to baselines, FID of our method are much lower and close to real dances
[Table]
which validates the good performance of our method on long-term generation.
5. A BLATION S TUDY
Comparison on Encoder Structures.
We compare the different encoder structures in our model with the same LSTM decoder and the same curriculum learning strategy, as shown in Table 3.
The transformer encoders outperform LSTM and the encoder in ConvS2S (Gehring et al., 2017) on both FID and style consistency, due to its superior performance on sequence modeling.
Although the transformer encoder with our proposed local self-attention slightly underperforms the global self-attention by 0.7 higher in FID and 1.5% lower in ACC, it has much fewer parameters in our settings.
This conﬁrms the effectiveness of the local self-attention on long sequence modeling.
Comparison on Learning Strategies.
To investigate the performance of our proposed learning approach, we conduct an ablation study to compare the performances of our model using different learning strategies.
As we can see the right of in Table 3, all curriculum learning strategies signiﬁcantly outperform the static auto-condition strategy (Li et al., 2017).
Among them, the linear growth function f ( t ) =
(cid:98) λt (cid:99) performs best, which might be that increasing the difﬁculty of curriculum too fast would result in the model degradation.
Besides, the original teacher-forcing strategy has the highest FID and the lowest style accuracy due to severe error accumulation problem, which indicates that using our proposed curriculum learning strategy to train the model can effectively alleviate this issue.
C ONCLUSION
In this work, we propose a novel seq2seq architecture for long-term dance generation with music, e.g., about one-minute length.
To efﬁciently process long sequences of music features, we introduce a local self-attention mechanism in the transformer encoder structure to reduce the quadratic memory requirements to the linear in the sequence length.
Besides, we also propose a dynamic auto-condition training method as a novel curriculum learning strategy to alleviate the error accumulation of autoregressive models in long motion sequence generation, thus facilitate the long-term dance generation.
Extensive experiments have demonstrated the superior performance of our proposed approach on automatic metrics and human evaluation.
In future work, we will explore the end-to-end method that could work with raw audio data instead of preprocessed audio features.
A PPENDIX 6. M ORE D ETAILS ABOUT D ATA C OLLECTION
The dance videos are collected from YouTube by crowd workers, which are all solo dance videos with 30FPS.
We trim the beginning and ending several seconds for each of collected videos to remove the silence parts, and split them into one-minute video clips.
Then, we extract 2D pose data from these video clips by OpenPose (Cao et al., 2017) with 15FPS setting and collect 790 one-minute clips of pose data with 15FPS, totaling about 13 hours.
Finally, we extract the corresponding audio data from video clips by FFmpeg 3 .
The code is implemented based on PyTorch framework.
MIT License will be used for the released code and data.
6. E XTRACTED M USIC F EATURES
In this section, we detail the extracted features of music that are feeded into our model.
Speciﬁcally, we leverage the public Librosa (McFee et al., 2015) to extract the music features including: 20dim MFCC , 20-dim MFCC delta , 12-dim chroma , 384-dim tempogram , 1-dim onset strength (i.e., envelope ) and 1-dim one-hot beat , as shown in Table 4.
[Table]
6. D OWNSTREAM A PPLICATIONS
The downstream applications of our proposed audio-conditioned dance generation method include: (1) Our model can be used to help professional people choreograph new dances for a given song and teach human how to dance with regard to this song; (2) With the help of 3D human pose estimation (Ci et al., 2019) and 3D animation driving techniques, our model can be used to drive the various 3D character models, such as the 3D model of Hatsune Miku (very popular virtual character in Japan).
This technique has the great potential for the virtual advertisement video generation that could be used for the promotion events on social medias like TikTok, Twitter and etc.
6. M USICAL B EAT D ETECTION UNDER L OW S AMPLING R ATES
In this section, we conduct an additional experiment to evaluate the performance of Librosa beat tracker under 15,400Hz sampling rate.
Speciﬁcally, we ﬁrst utilize it to extract the onset beats from audio data with 15,400Hz and 22,050Hz respectively, then compare two groups of onset beats under the same time scale.
Since 22,050Hz is a common sampling rate used in music information retrieval, we deﬁne the beat alignment ratio as B 2 /B 1 , where B 1 is the number of beats under 15,400Hz and
B 2 is the number of beats under 15,400Hz that are aligned with the beats under 22,050Hz.
One beat is counted when | t 1 − t 2 |≤ ∆ t .
t 1 denotes the timestamp of a certain beat under 15,400Hz while trefers to the timestamp of a certain beat under 22,050Hz, ∆ t is the time offset threshold.
In the experiment, we set ∆ t = 1 / 15 s (FPS is 15) and calculate the beat alignment ratio for audio data of three styles respectively.
We randomly sample 10 audio clips from each style to calculate the beat alignment ratio and do the sampling for 10 times to take the average.
As shown in Table 5, most of beats extracted with 15,400Hz are aligned with those extracted with 22,050Hz.
Besides, we randomly sample an audio clip and visualize beat tracking curves of its ﬁrst 20 seconds under the sampling rates of 15,400Hz and 22,050Hz.
As we can see in Figure 5, most of the beats from 15,400Hz and 22,050Hz are aligned within the time offset threshold ∆ t = 1 / 15 s .
[Figure]
[Table]
[Figure]