Paper: Duolando: Follower GPT with Off-Policy Reinforcement Learning for Dance Accompaniment (ICLR-2024)

Reviewer comment (Experiments):
To gain a better understanding of the differences between various dance styles within the dataset and across datasets, it would be beneficial to provide more data statistics, such as movement speed and action distribution. More details on data processing would be helpful to understand it.

Author response:
**A1**: We compute the statistics of speed distributions for different types of dances. This part is revised in Appendix B.1 (pp.14).

Reviewer comment (Presentation):
In Equation 1, it's stated that both p and M are inputs, but in Figure 3, only p is shown as an input.

Author response:
**A2**: The VQ-VAE is designed to encode $p$ and subsequently decode it into both $p$ and $M$. $p$ and $M$ in Equation 1 are ground truth targets in the learning, not inputs.

Reviewer comment (Experiments):
It would be interesting to explore which signal, between music and leader motion, has a more dominant influence. For instance, when swapping the music between two test cases in the test set, and using music(B) with leader motion(A) and music(A) with leader motion(B) as inputs, how does this affect the results?

Author response:
**A3**: As required by the reviewer, we randomize music during the testing phase. The experimental results are shown in Table 4 （pp.17 Appendix D.1). For “Duolando w/o. RL”, incorrect music input causes a reduction in all motion metrics and music metrics. Specifically, the BAS decreases by 12%, while the Beat Echo Degree (BED) decreases relatively less (8%), indicating that leader motion is more crucial in our proposed framework.

Reviewer comment (Writing):
There is some ambiguity in the terminology between "duet dance generation" and "dance accompaniment." It's important to differentiate them more clearly in the draft. Additionally, it's worth noting that this dataset potentially holds promise for duet dance generation, where only music serves as input to generate movements for two individuals.

Author response:
**A4**: We appreciate the reviewer indicating that. We have revised our paper.

Reviewer comment (Writing):
How is music represented? Midi or spectrogram or raw waveform? I didn’t see where the paper defined the representation of it.

Author response:
**A1**: We apologize for the oversight.

*The music features $\bf{\it m}$ are extracted from the audio signal using the public audio processing toolbox Librosa (McFee et al. 2015), including mel frequency cepstral coefficients (MFCC), MFCC delta, constant-Q chromagram, onset strength and onset beat, which are 54-dimension in total, and are mapped to the same dimension （438）of GPT via a learned linear layer.*

This part was originally in the main text but was inadvertently omitted when moving it to the appendix due to page limitation. We have added this back in the revised version (pp.15 Appendix B.2).

Reviewer comment (Experiments):
An important ablation study is missing, what if the music is not fed as input. I think it still makes sense to generate a follower dancer as the rhythm of music is already embedded in the leader dancer?

Author response:
**A2**: (1) To explore the effect of the music signal input, we train a new variant GPT without the slots for music: ***Duolando w/o. RL*** $m$. The experimental results have been updated in the revised draft (Table 4, Appendix D.1, pp. 18). Without the music input, the follower achieves a comparable Beat-Align Score (BAS) to that with music input since the follower is explicitly anchored to the leader's position and will follow the rhythm indirectly.

(2) The above alignment is under the condition that the follower's motion aligns with the music well. In the other experiment, we randomized the music input in the test phase. As shown in Table 4, the BAS of ***Duolando w/o. RL*** $m$ is 0.1657, lower than that with music input (0.1840), which shows the music encoding will significantly impact the motion when music is not aligned with the movement.

Reviewer comment (Experiments):
The RL setup aims to handle the OOD data for unseen dance motions, and the authors present an off-policy learning approach. However, I didn’t see ablations against on-policy RL nor ablations on the reward design, which looks very ad-hoc to me.

Author response:
**A3**: (1) We present the reward curves of the proposed off-policy RL and the on-policy AC (Siyao et *al.*, 2022) on test set in Figure 8, pp.18 and Appendix D.2, pp. 19. The average sequantial reward for the off-policy RL shows a steady increase; in contrast, the on-policy RL trend is less stable and even collapses at around the 30-th epoch. This aligns with the phenomena reported in (Siyao et *al.*, 2022) that excessive fine-tuning in on-policy AC leads to instability.

(2) Given that our reward system comprises only a single reward, conducting a compositional ablation study is impractical. On the reward itself,  we introduced a threshold of for the penalty. If without this threshold, the follower tend to perform excessively redundant small steps to align the translation strictly.

Reviewer comment (Evaluation):
The supplementary material provides videos for original dataset and generated samples. However, I find some samples where actions are not well-aligned with music beats at all, I doubt whether it is something performed by professional dancers.

Author response:
**A4**: All dancers involved in our recordings are professionally trained and have won awards in professional competitions. After a thorough comparison, we confirm that the segments provided in our demo align with the live video footage. The perception of misalignment between motion and music that the reviewer notes might be attributed to the concept of 'rubato'. In ballroom dancing, particularly in smooth styles, dancers may intentionally delay a step or movement to coincide with a specific musical element or to create a desired effect or expression. This artistic choice enhances the performance's expressiveness but might appear as a misalignment to those unfamiliar with the technique.

Reviewer comment (Evaluation):
From the generated samples, I saw that sometimes part of a body model passing through part of another body model in an unnatural manner.

Author response:
**A5:** Indeed. As the first solution to dance accompaniment, while our method is not yet perfect, it effectively tackled some key issues through the incorporation of Interaction Coordination (IC), relative translation (tr), and Reinforcement Learning (RL). These improvements resolve issues like action irrelevance, inappropriate relative displacement, and follower's lower body incompatibility, making our approach significantly superior to existing solo dance methods in the interactive scenarios. It can serve as a solid baseline for this newly proposed task.

Reviewer comment (Experiments):
I found a major weakness in the off-policy RL part. For a pure kinematics-based method, there might exist a couple of artifacts, such as floor-skating, penetration between bodies and ground, and other physical-infeasible dynamics. The videos in the supplementary materials also showed such artifacts. The paper only addresses the problem of misalignment of lower-body motion and full-body motion; why the author just stopped here? It seems that other artifacts might also be alleviated through similar RL tuning. Has the author tried to address such artifacts? Does the author believe such off-policy tuning combined with some heuristic reward function can reduce most artifacts?

Author response:
**A1**: We appreciate the reviewer's suggestions for a more in-depth exploration of the RL rewards to enhance dance quality.

(1) Our primary focus in this work is to enable the follower to dance responsively to the leader. Beginning with a solo dance GPT structure similar to *Bailando* (Siyao et *al.*, 2022), we employ Interaction Coordination (IC) and explicit relative translation (***tr***) to make the follower correspond with the leader on movement and positioning, respectively. The proposed RL concentrates on solving/suppressing the side effect of ***tr*** -- the skating effect, caused by misalignment between lower body and global shift.


Regarding penetration issues, potential post-processing exists in the industry and the academy [1,2,3]. A feasible RL solution can be rewarding non-intersection while penalizing the corresponding intersecting parts based on penetration depth. For instance, intersections on hands and arms should lead to a penalty on the upper body, while that on the trunk should impact translation (***tr***).

(2)  We acknowledge that RL alone may not resolve all issues. In practical applications, graphical post-processing is expected for further quality enhancement.

#### Reference


[1] Smith et *al.* Open dynamics engine, 2005

[2] Pavlakos et *al.*. Expressive body capture: 3d hands, face, and body from a single image. CVPR 2019.

[3] Müller et *al.* On self-contact and human pose. CVPR 2021.

Reviewer comment (Evaluation):
The author mentioned using off-policy RL to reduce the artifacts of misalignment of lower and full-body motion. It would be beneficial to add some new metrics to evaluate this specific improvement.

Author response:
**A2**: To show effect of RL on reducing skating effect, we propose a metric named *slipping ratio* (SR). Specifically, we compute the percentation of the length where the feet/legs do not move obviously but the global shift is significant (> 1 cm/frame). The results are shown in Table 4, Appendix D.3, pp.19. Experimental results show our proposed RL can reduce over 60% of skating in that without RL.

Reviewer comment (Experiments):
Why use 2D skeletons for user study? Why not use 3D-rendered motions, like the one in the demo video?

Author response:
**A3**: As required by the reviewer, we conduct a new user study using 3D-rendered SMPL-X models. We also updated the results on the revised draft. Generally, the average winning rates do not change greatly and do not change the conclusion.

Reviewer comment (Evaluation):
What is the training time for each of the stages?

Author response:
**A4**: The training time for VQ-VAEs is about 2 days. The training time of GPT in the supervised learning stage is around 1.5 days and that of the RL stage is about 18 hours. Four NVIDIA V100 GPUs are used for training.

Reviewer comment (Evaluation):
A general question is how to evaluate the quality of the motion-capture? I watched the video, and it seems there are some flickering frames and some exaggerated hand motions.

Author response:
**A5**: Thanks for bringing up the issues with the mocap evaluation. We have observed similar phenomena in our data and have undertaken methods to refine mocap data quality along two dimensions:

1. **Flickering frames**: Inaccuracies arise when marker points are potentially obscured during the collection process, resulting in inaccuracies in specific frames. To address this, we introduce two kinds of information: (1) **Single-frame confidence**. We leverage marker point confidence during the fitting process, excluding points with low confidence. (2) **Temporal smoothness.** If most of the markers within a frame have low confidence, the fitting outcome may be suboptimal. We introduce temporal information in post-processing to first detect abrupt changes from the temporal axis. Subsequently, we eliminate such frames and employ interpolation to complete the missing frames.
2. **Exaggerated hand motion**: This issue typically arises from either accumulated errors in the original glove data or a lack of constraint information for joints. To avoid this, explicit constraints are imposed on each finger and wrist joint according to human anatomy. For instance, finger joints are restricted from undergoing self-rotation, and hand wrists are also constrained against self-rotation.).

Reviewer comment (Reproducibility):
Perhaps moreso than the proposed methods, the DD100 dataset may be the most valuable aspect of this work to the dance generation community. This paper promises to release “MP3s” associated with the dataset but fails to report details about the copyright status of the music in the dataset. Listening to the demo videos, the dataset appears to feature copyrighted material (e.g. “Charlie Puth - LA Girls”). It is likely that, even if the authors release the audio, they will likely be forced to take it down eventually, compromising the value of the dataset to the research community.

Author response:
**A2**: **Choice of Music.** These tracks were chosen by the performers from their familiar collections to ensure the quality of performance.

**‘Fair Use’ [1, 2] Justification.** The music segments are modified (clipped, and partially slowed down), deviating from their original form. These modifacations, coupled with their use solely for model training/testing rather than entertainment, comply with fair use criteria. Additionally, the academic nature of our work and the absence of public searchability (we won’t provide music info beyond the audio) avoid potential commercial impact. Therefore, the music included in our dataset falls under ‘fair use’ provisions, while it also align with previous practices [3, 4] in dance geration domain, presenting a relatively low risk. We have also updated these facts in the ethical statement in our paper.

**Contingency Plans.** Should the need arise to remove certain tracks, we can provide them in a non-playable, digitized format suitable for model training/testing. Or at least, we can offer only the titles and relevant details (like YouTube links and specific segment/speed information) with automatically downloading/processing scripts, a common method used in prior academic datasets [5, 6].

#### Reference

[1]https://guides.nyu.edu/fairuse#:~:text=Fair%20use%20allows%20limited%20use,automatically%20qualifies%20as%20fair%20use

[2] https://en.wikipedia.org/wiki/Fair_use

[3] Lee et *al.*, Dancing to Music, NeurIPS2019

[4] Huang and Hu et *al.*, Dance Revolution: Long-Term Dance Generation with Music via Curriculum Learning, ICLR 2021

[5] Siarohin et *al.*, First Order Motion Model for Image Animation, NeurIPS 2019

[6] Miao et *al.*, VSPW: A Large-scale Dataset for Video Scene Parsing in the Wild, CVPR 2021

Reviewer comment (Novelty):
There is not a lot of information in this paper that would be of interest to researchers outside of the dance generation community. Perhaps there is something reusable happening in the use of RL to refine GPT models, but this is only explored within the context of dance generation. Though ICLR does occasionally have dance generation work in its proceedings, I suspect that this paper overall will not be particularly interesting to the broader ICLR community.

Author response:
**A3**: Our method can extend beyond dance generation and offer insights for human-human interaction tasks, a key area of 3D human motion and virtual reality. Additionally, it holds promise for integrating 3D human interaction within unified language models (GPT), which could interest a broader range of researchers. 

Furthermore, our proposed DD100 dataset contributes strong interaction data with long average durations to the human-human interaction research community.