swdq/FlowerDance-Shorts

FlowerDance Shorts — epoch 850

音源全体から1人分のダンスを生成する、FlowerDanceの追加学習モデルです。 127本の動画から作った学習データで、390 epochのモデルを起点に追加学習しました。 公開している重みは 850 epoch の保存済みモデルです。学習は853 epochの完了後に停止しました。

音源から推論する / Live demo

Contents

• model.safetensors: portable model weights, exported exactly from the epoch 850 checkpoint.
• config.json: architecture and sampling settings.
• normalizer.json: motion scaling parameters; no Python pickle required for inference.
• infer_safetensors.py: full-audio inference entry point.
• model/, flowerdance/, dataset/, vendor/: the Python source used by this adaptation.
• examples/: four audio-bearing comparison videos and generated NPZ pose files.
• evaluation/: training logs and metrics for the two selected training examples.
• training_summary.json: provenance and SHA-256 hashes.

The optimizer and RNG state are not included in this inference export. This is not a Transformers AutoModel checkpoint.

Inference

The ZeroGPU Space accepts audio uploads (1 second to 6 minutes), processes the complete sequence, and returns an audio-bearing front/side skeleton video and motion NPZ. It uses the epoch 850 weights, with 40 Euler steps by default. The comparison gallery remains available in a separate tab.

The tested environment is NVIDIA GB10 (DGX Spark), Python 3.12 and PyTorch 2.14 / CUDA 13. A CUDA-compatible PyTorch installation and a compatible Mamba selective_scan_cuda extension are required. The native extension is architecture-specific and is intentionally not bundled as an ARM64-only binary. The live ZeroGPU Space also runs with Python 3.12, PyTorch 2.9.1 and the matching x86_64 Mamba wheel.

Download this repository, install CUDA-enabled PyTorch for your environment, and install the dependencies:

hf download swdq/FlowerDance-Shorts --local-dir FlowerDance-Shorts
cd FlowerDance-Shorts
pip install -r requirements-inference.txt
pip install mamba-ssm --no-build-isolation
# Install ffmpeg using your OS package manager.
PYTHONPATH="$PWD/vendor${PYTHONPATH:+:$PYTHONPATH}" python infer_safetensors.py \
  --model-dir . --audio input.wav --output motion.npz --steps 40

Mamba installation may require a matching CUDA toolkit/compiler. The vendored Python code uses the installed native extension. The safetensors export is checked for exact tensor equality with the original checkpoint; full-audio inference is also checked locally.

Model and output

• Base: xlt99/FlowerDance, from XulongT/FlowerDance.
• BiMamba decoder: latent dimension 512, 8 dance layers, 4 music layers.
• Music: 35 channels at 30 Hz, extracted from the full audio.
• Motion: 151 channels (4 contact channels + root XYZ + 24 joint rotations in 6D).
• Fine-tuning objective: rectified-flow velocity prediction, interval zero, plus FK/velocity/acceleration losses.
• Sampling: 40 Euler steps, seed 1234, internal genre ID 5. No genre input is required from the user.
• Each step processes the whole sequence. seq_len=1200 is a constructor setting, not a 40-second output cap.
• motion.npz contains raw 30 Hz motion and Y-up root-relative pose at 60 Hz (frames repeated solely for playback).

The comparison videos show the source video on the left and generated skeletons from the front and side. Training examples also show the teacher skeleton used in fine-tuning. Source videos are only displayed for comparison; generation is conditioned on audio alone.

Evaluation and limitations

Two of the 103 added training clips were selected using the epoch 420 fixed-noise flow-loss ranking:

Example	Selection	Epoch 850 root-relative MPJPE	Motion / target	Maximum joint displacement per frame
マリ箱ダンス	Median loss	2.94 cm	0.995	0.227 m
新宝島踊ってみた	Highest loss	3.28 cm	0.952	1.062 m

These are training-set reconstruction comparisons, not held-out accuracy. The average error can hide individual jumps: the 新宝島 example contains abrupt joint movement. A low loss does not establish realistic or correct choreography on new music.

Training targets are pseudo-labels from SAM3 Body/MHR, mapped to a 24-joint skeleton and converted using approximate inverse kinematics. They are not manually annotated ground truth or licensed SMPL fits. Global travel and joint twist are not accurately recovered. The model produces one dancer, even if a source video contains multiple people.

Comparison sources and full metrics are listed in examples/manifest.json. The source music and videos retain their respective ownership; this model publication does not grant rights to those media. No additional license is asserted for the upstream FlowerDance weights or code. The vendored Mamba code retains its existing license at vendor/MAMBA_LICENSE.

Fuwawa MMD preview

The Space retargets generated body motion to Fuwawa Abyssgard and plays a 3D character preview synchronized to the audio. The browser can orbit and zoom the view. The initial preview is the Maribako training example; new inference results replace it.

Rendered dance, source/pose comparison, and VMD are available for that 448-frame example. The 24-joint motion drives 19 body bones with rest-pose calibration and a center translation. Fingers, expressions, and secondary hair/cloth dynamics are not predicted. The VMD disables leg IK to preserve the baked FK motion.